Icinga / Icinga/icinga2

Icinga2 service coredumps on master during config reload due to OTLPMetricsWriter

Open
#11,010 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/distributed area/opentelemetry
Dominant language
C++
Stars
2.2k
Forks
616
Avg merge
5d 6h
Merged PRs (30d)
10

Description

Describe the bug

icinga2 service coredumps sometimes on config reload due to OTLPMetricsWriter, writing to a victoriametrics instance.

To Reproduce

Not sure how to just reload the config and sometimes like 1/50 times the service will coredump

Expected behavior

Icinga2 service should not coredump :)

Your Environment

  • Version used (icinga2 --version):
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.16.4-1)

  • Operating System and version:
    Rocky Linux 9.8 Blue Onyx

  • Enabled features (icinga2 feature list):
    Enabled features: api checker icingadb mainlog notification otlpmetrics

  • Icinga Web 2 version and modules (System - About):
    Loaded Libraries
    icinga/icinga-php-library 1.0.0
    icinga/icinga-php-thirdparty 1.0.0
    icinga/icinga-php-legacy 1.1.0

Loaded Modules
icingadb 1.4.0
businessprocess 2.6.0
incubator 0.23.0
oidc 0.7.7
pdfexport 0.13.0
perfdatagraphs 0.5.0
perfdatagraphsprometheus 0.2.1
reporting 1.1.0
vspheredb 1.8.5
x509 1.4.0

  • Config validation (icinga2 daemon -C):
    icinga2 daemon -C
    [2026-08-28 08:15:33 +0200] information/cli: Icinga application loader (version: r2.16.4-1)
    [2026-08-28 08:15:33 +0200] information/cli: Loading configuration file(s).
    [2026-08-28 08:15:35 +0200] information/ConfigItem: Committing config item(s).
    [2026-08-28 08:15:35 +0200] information/ApiListener: My API identity: redacted
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 518 Notifications.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 469 HostGroups.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 3100 Hosts.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 EventCommand.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 2570 Downtimes.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 227 Comments.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 FileLogger.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 9 Zones.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 16 Endpoints.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 5 ApiUsers.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 ApiListener.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 OTLPMetricsWriter.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 627 CheckCommands.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 60 TimePeriods.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 19 Users.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 2 UserGroups.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 63694 Services.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1 ServiceGroup.
    [2026-08-28 08:15:40 +0200] information/ConfigItem: Instantiated 1436 ScheduledDowntimes.
    [2026-08-28 08:15:40 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
    [2026-08-28 08:15:40 +0200] information/cli: Finished validating the configuration file(s).

  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
    Running 7 satellite zones with 2 satellites each + master zone with 2 masters in HA.

Additional context

coredumpctl info

           PID: 2026052 (icinga2)
           UID: 989 (icinga)
           GID: 989 (icinga)
        Signal: 11 (SEGV)
     Timestamp: Fri 2026-08-28 07:45:17 CEST (32min ago)
  Command Line: /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
    Executable: /usr/lib64/icinga2/sbin/icinga2
 Control Group: /system.slice/icinga2.service
          Unit: icinga2.service
         Slice: system.slice
       Boot ID: 5af60cf96e624cac98b79f7537f3fd51
    Machine ID: 47ea5d55c769484ab7fa48ead7d0e986
      Hostname: redacted
       Storage: /var/lib/systemd/coredump/core.icinga2.989.5af60cf96e624cac98b79f7537f3fd51.2026052.1787895917000000.zst (present)
  Size on Disk: 124.7M
       Message: Process 2026052 (icinga2) of user 989 dumped core.

                Stack trace of thread 2035197:
                #0  0x0000000000d4aad1 _ZN6icinga4OTel5StartEv (icinga2 + 0x94aad1)
                #1  0x0000000000da7d6a _ZN6icinga17OTLPMetricsWriter6ResumeEv (icinga2 + 0x9a7d6a)
                #2  0x000000000079a230 _ZN6icinga12ConfigObject12SetAuthorityEb (icinga2 + 0x39a230)
                #3  0x00000000008d0e89 _ZN6icinga11ApiListener21UpdateObjectAuthorityEv (icinga2 + 0x4d0e89)
                #4  0x00000000008bb9c5 _ZN6icinga11ApiListener15HelloAPIHandlerERKN5boost13intrusive_ptrINS_13MessageOriginEEERKNS2_INS_10DictionaryEEE (icinga2 + 0x4bb9c5)
                #5  0x00000000009117f9 _ZNSt17_Function_handlerIFN6icinga5ValueERKN5boost13intrusive_ptrINS0_13MessageOriginEEERKNS3_INS0_10DictionaryEEEEPSC_E9_M_invokeERKSt9_Any_dataS7_SB_ (icinga2 + 0x5117f9)
                #6  0x00000000008f7aaa _ZN6icinga17JsonRpcConnection14MessageHandlerERKN5boost13intrusive_ptrINS_10DictionaryEEE (icinga2 + 0x4f7aaa)
                #7  0x00000000008f855d _ZN6icinga17JsonRpcConnection22HandleIncomingMessagesEN5boost4asio19basic_yield_contextINS2_15executor_binderIPFvvENS2_8executorEEEEE (icinga2 + 0x4f855d)
                #8  0x000000000091b992 _ZZN6icinga8IoEngine14SpawnCoroutineIN5boost4asio10io_context6strandEZNS_17JsonRpcConnection5StartEvEUlNS3_19basic_yield_contextINS3_15executor_binderIPFvvENS3_8executorEEEEEE_EEvRT_T0_ENKUlSD_E_clESD_.lto_priv.0 (icinga2 + 0x51b9>
                #9  0x000000000099eb32 _ZN5boost10coroutines6detail20trampoline_push_voidINS1_21push_coroutine_objectINS0_14pull_coroutineIvEEvRNS_4asio6detail16coro_entry_pointINS6_15executor_binderIPFvvENS6_10io_context6strandEEEZN6icinga8IoEngine14SpawnCoroutineISD_>
                #10 0x00007f8d2ecb824f make_fcontext (libboost_context.so.1.75.0 + 0x124f)
                #11 0x0000000000000000 n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64

           PID: 2026052 (icinga2)
           UID: 989 (icinga)
           GID: 989 (icinga)
        Signal: 11 (SEGV)
     Timestamp: Fri 2026-08-28 07:45:17 CEST (32min ago)
  Command Line: /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
    Executable: /usr/lib64/icinga2/sbin/icinga2
 Control Group: /system.slice/icinga2.service
          Unit: icinga2.service
         Slice: system.slice
       Boot ID: 5af60cf96e624cac98b79f7537f3fd51
    Machine ID: 47ea5d55c769484ab7fa48ead7d0e986
      Hostname: redacted
       Storage: /var/lib/systemd/coredump/core.icinga2.989.5af60cf96e624cac98b79f7537f3fd51.2026052.1787895917000000.zst (present)
  Size on Disk: 124.7M
       Message: Process 2026052 (icinga2) of user 989 dumped core.

                Stack trace of thread 2035197:
                #0  0x0000000000d4aad1 _ZN6icinga4OTel5StartEv (icinga2 + 0x94aad1)
                #1  0x0000000000da7d6a _ZN6icinga17OTLPMetricsWriter6ResumeEv (icinga2 + 0x9a7d6a)
                #2  0x000000000079a230 _ZN6icinga12ConfigObject12SetAuthorityEb (icinga2 + 0x39a230)
                #3  0x00000000008d0e89 _ZN6icinga11ApiListener21UpdateObjectAuthorityEv (icinga2 + 0x4d0e89)
                #4  0x00000000008bb9c5 _ZN6icinga11ApiListener15HelloAPIHandlerERKN5boost13intrusive_ptrINS_13MessageOriginEEERKNS2_INS_10DictionaryEEE (icinga2 + 0x4bb9c5)
                #5  0x00000000009117f9 _ZNSt17_Function_handlerIFN6icinga5ValueERKN5boost13intrusive_ptrINS0_13MessageOriginEEERKNS3_INS0_10DictionaryEEEEPSC_E9_M_invokeERKSt9_Any_dataS7_SB_ (icinga2 + 0x5117f9)
                #6  0x00000000008f7aaa _ZN6icinga17JsonRpcConnection14MessageHandlerERKN5boost13intrusive_ptrINS_10DictionaryEEE (icinga2 + 0x4f7aaa)
                #7  0x00000000008f855d _ZN6icinga17JsonRpcConnection22HandleIncomingMessagesEN5boost4asio19basic_yield_contextINS2_15executor_binderIPFvvENS2_8executorEEEEE (icinga2 + 0x4f855d)
                #8  0x000000000091b992 _ZZN6icinga8IoEngine14SpawnCoroutineIN5boost4asio10io_context6strandEZNS_17JsonRpcConnection5StartEvEUlNS3_19basic_yield_contextINS3_15executor_binderIPFvvENS3_8executorEEEEEE_EEvRT_T0_ENKUlSD_E_clESD_.lto_priv.0 (icinga2 + 0x51b992)
                #9  0x000000000099eb32 _ZN5boost10coroutines6detail20trampoline_push_voidINS1_21push_coroutine_objectINS0_14pull_coroutineIvEEvRNS_4asio6detail16coro_entry_pointINS6_15executor_binderIPFvvENS6_10io_context6strandEEEZN6icinga8IoEngine14SpawnCoroutineISD_ZNSF_17JsonRpcConnection5StartEvEUlNS6_19basic_yield_contextINS9_ISB_NS6_8executorEEEEEE_EEvRT_T0_EUlSM_E_EENS0_30basic_standard_stack_allocatorINS0_12stack_traitsEEEEEEEvNS_7context6detail10transfer_tE.lto_priv.0 (icinga2 + 0x59eb32)
                #10 0x00007f8d2ecb824f make_fcontext (libboost_context.so.1.75.0 + 0x124f)
                #11 0x0000000000000000 n/a (n/a + 0x0)
                ELF object binary architecture: AMD x86-64

GNU gdb (Rocky Linux) 16.3-3.el9
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/icinga2/sbin/icinga2...
Reading symbols from /usr/lib/debug/usr/lib64/icinga2/sbin/icinga2-2.16.4-1.el9.x86_64.debug...

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 2035197]
[New LWP 2035202]
[New LWP 2026059]
[New LWP 2026055]
[New LWP 2026060]
[New LWP 2035195]
[New LWP 2035204]
[New LWP 2026723]
[New LWP 2026057]
[New LWP 2035199]
[New LWP 2026058]
[New LWP 2026053]
[New LWP 2026054]
[New LWP 2026056]
[New LWP 2026052]
[New LWP 2035201]
[New LWP 2035205]
[New LWP 2035198]
[New LWP 2035200]
[New LWP 2035203]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  std::__atomic_base<bool>::exchange (__m=<optimized out>, __i=<optimized out>, this=<optimized out>, this=<optimized out>, __i=<optimized out>, __m=<optimized out>) at /usr/include/c++/11/bits/atomic_base.h:503
503           exchange(__int_type __i,
[Current thread is 1 (Thread 0x7f8d2b55c640 (LWP 2035197))]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the OTLPMetricsWriter::Resume and OTel::Start frames in the supplied coredumpctl and GDB output, then examine what happens during a configuration reload and authority update. Reproduce the intermittent reload failure if possible and use the core dump to identify the crash condition; done means reloads no longer produce a SIGSEGV.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.