Check execution stalls during configuration reload and bursts afterwards on large installation

Describe the bug

On a large Icinga 2 installation, a configuration deployment/reload causes a strong reduction of check execution while the new configuration is being validated, followed by a very large catch-up burst once the new Icinga process takes over.

During the analyzed reload, the validation process itself consumed tens of CPU cores. Check execution was strongly reduced for roughly 38 seconds and then increased from a normal baseline of about 189 process executions/s to a peak of about 1167 executions/s. At the same time, overall CPU usage reached about 99% and the run queue peaked around 246.

This appears inconsistent with the documented reload behavior, where configuration validation is performed in a separate process and should not stop other events such as check execution and notifications.

We are not currently observing a violation of MaxConcurrentChecks: it is set to 512 and, so far, we have not measured more than 512 concurrently running checks. The issue appears to be related to check scheduling/spreading across the reload rather than to the concurrency limit itself.

To Reproduce

On this installation, the behavior is reproducible during regular Icinga Director deployments that trigger a master configuration validation and reload.

1. Start from the normally running Icinga master with checks distributed over time.
2. Deploy a configuration change from Icinga Director.
3. Observe the separate Icinga configuration-validation process and the check execution rate during the validation/reload window.
4. Once the new Icinga process takes over, observe the concentrated burst of check executions and the resulting CPU/run-queue saturation.

The observed sequence is approximately:

normal check execution
        |
        v
Director deployment
        |
        v
configuration validation process starts
        |
        v
check execution strongly decreases / nearly stalls
        |
        v
old/new Icinga processes overlap during reload
        |
        v
new process becomes active
        |
        v
large catch-up burst of checks
        |
        v
CPU and run queue saturation
        |
        v
system gradually returns to normal

During the analyzed deployment, eBPF and process-level sampling showed approximately:

• baseline Icinga CPU consumption: ~1.9 CPU cores
• configuration validation:
  • average: ~15.3 CPU cores
  • peak: ~26.7 CPU cores
• old/new process overlap:
  • average: ~15.9 CPU cores
  • peak: ~37.5 CPU cores
• new main Icinga process:
  • observed peak: ~34.7 CPU cores
• post-reload period:
  • average: ~13.5 CPU cores
  • peak: ~32.5 CPU cores
• process execution baseline: ~189 execs/s
• process execution peak after reload: ~1167 execs/s
• system CPU busy: up to ~99%
• run queue peak: ~246
• roughly 38 seconds where check execution was strongly reduced before the catch-up burst

During the critical validation/reload phase, Icinga processes accounted for roughly 94-98% of the observed CPU usage.

We can provide the timestamped eBPF capture and more detailed per-second data if useful.

Expected behavior

According to the Icinga documentation, a reload validates the configuration in a separate process and should not stop other events such as check execution and notifications.

We therefore expect the currently running Icinga process to continue scheduling/executing checks normally while the new configuration is validated.

When the new process takes over, we would also expect check scheduling to remain reasonably spread over time rather than causing a very large immediate catch-up burst of checks that saturates the host.

Some additional CPU usage during validation is expected on a large configuration. The unexpected part is the combination of:

1. very high validation CPU usage,
2. a strong reduction of check execution during the reload window,
3. and the large concentrated burst immediately afterwards.

Screenshots

No screenshots attached yet.

The investigation was performed primarily with timestamped eBPF/process data. We can attach graphs or the raw capture if this is useful for further analysis.

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):
  • Icinga binary reports: r2.15.6-1
  • Vendor package: icinga2-2.15.6_neteye1.73.1-1.el8.x86_64
  • Repository: NetEye
  • Previous installed package was icinga2-2.15.5_neteye1.73.0-2.el8.x86_64
  • We have not yet established whether the same behavior was present with 2.15.5.
• Operating System and version:
  • Red Hat Enterprise Linux 8.10 (Ootpa)
  • Kernel: 4.18.0-553.148.1.el8_10.x86_64
  • 48 logical CPUs
• Enabled features (icinga2 feature list):
  • api
  • checker
  • icingadb
  • influxdb
  • livestatus
  • mainlog
  • notification
• Icinga Web 2 version and modules (System - About):
  • Icinga Web 2 package: 2.12.4_neteye1.158.0
  • Icinga Director package: 1.11.4_neteye1.41.0
• Config validation (icinga2 daemon -C):
  • We have not run an additional standalone validation specifically for this report because configuration validation itself is part of the performance issue being investigated on this production system.
  • Director-triggered deployments complete their configuration validation successfully before the reload.
  • We can provide the output of a controlled standalone validation if requested.
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes:
  • This is a NetEye installation using its master-instance layout. The affected master instance runs through icinga2-master.service.
  • The NetEye icinga2-master CLI command is only a wrapper around the regular icinga2 binary. It loads the instance-specific sysconfig and passes ConfigDir, DataDir, LogDir, CacheDir, SpoolDir, and InitRunDir as constants.
  • We can provide a sanitized zones.conf / endpoint and zone listing if required.

Relevant runtime values:

MaxConcurrentChecks = 512
Configuration.Concurrency = not explicitly configured

Configuration.Concurrency is not explicitly set. In upstream Icinga 2, when it is not modified, it defaults to std::thread::hardware_concurrency(). On this host, nproc reports 48 logical CPUs.

Additional context

Historical issue that may be related

We found the historical issue:

https://github.com/Icinga/icinga2/issues/4277

"many check commands executed at same time when master reload"

That issue also involved a concentration of check executions around a master reload and was fixed as a race condition related to concurrent_checks.

We are not claiming this is the same bug.

In particular, unlike the historical issue, we currently have no evidence that MaxConcurrentChecks is being exceeded. We mention it because the temporal behavior around reload appears conceptually similar.

Plugin-side load was investigated separately

One particularly expensive family of approximately 2784 service checks was investigated separately.

Before optimization, those checks had an average execution time of about 1.28 seconds. After changing the implementation to use a per-host cache instead of repeatedly scanning a large JSON file, the average execution time dropped to about 0.05 seconds.

This significantly reduced plugin-side overhead, but it did not explain the configuration-validation CPU consumption or the check-execution behavior around reload. During the critical validation/reload phase, the Icinga processes themselves still dominated CPU usage.

Questions

We would appreciate clarification on the following:

1. During a configuration reload, should the old Icinga process continue scheduling and executing checks normally for the entire configuration-validation period?
2. Is a significant reduction or temporary near-stall of check execution during validation expected on a large configuration?
3. How are checks whose next_check becomes due during the validation/reload window handled when the new process takes over?
4. Is there a mechanism intended to preserve/re-spread their scheduling, or is an immediate catch-up burst expected?
5. Could this behavior indicate a scheduling/reload issue even though MaxConcurrentChecks itself is not exceeded?
6. Would you recommend explicitly benchmarking/lowering Configuration.Concurrency on a 48-CPU system before investigating the scheduler behavior further?
7. Are there specific debug logs, internal statistics, or eBPF probes that would be most useful to capture during another reload?

Start by reproducing a Director-triggered deployment while collecting the timestamped eBPF/process data described in the report, and compare it with a standalone icinga2 daemon -C validation. Review historical issue #4277 and investigate how check scheduling behaves during the old/new process overlap. Done means determining whether the stall and catch-up burst are expected and identifying a supported correction or clear diagnosis.