Icinga sends notifications for hosts a second after getting into soft state (1 out of 3 tries)

Hello everybody,

Some bit of context on the issue my team is facing.
We have an Icinga based environment set up on Kubernetes consisting of 2 masters and 2 satellites.
Our environment involves a down-up communication model, meaning that the agents (monitored VMs) connect to the satellites and the satellites connect to the masters.
The host template we are using for the monitored VMs is the following:
{ "accept_config": true, "check_command": "cluster-zone", "check_interval": "120", "max_check_attempts": "3", "retry_interval": "60", "enable_active_checks": true, "enable_flapping": true, "enable_passive_checks": false, "enable_perfdata": false, "has_agent": true, "master_should_connect": false, "object_type": "template", "vars": { "entity_of": "", "entity_type": "", "subscriptions": [ "INIT" ] }, "volatile": false }'

Our notification object is configured as:
{ "apply_to": "host", "assign_filter": "host.vars.team=%22MyTeam%22&host.zone=%22satellite%22", "imports": [ "template_mail-host-notification" ], "object_name": "mail-host-notification", "object_type": "apply", "period": "24x7", "states": [ "Down", "Up" ], "types": [ "Acknowledgement", "DowntimeEnd", "DowntimeRemoved", "DowntimeStart", "FlappingEnd", "FlappingStart", "Problem" ], "users": [ "my.user" ], "notification_interval": "3600", "times_begin": "0" }

Describe the bug

We have a CI/CD pipeline that updates or enforces the configuration to the Icingaweb Director component.
When applying the Director configuration, a few hosts changes their states to DOWN, but they get into a soft state first.
Although our host configuration implies max_check_attempts being set to 3, sometimes Icinga sends notifications for these hosts exactly 1 second after running the first check (see the screenshots).

To Reproduce

The issue at hand is not reproducible at every Director apply.

Expected behavior

Icinga to send notification when the object gets into Hard state.

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): v2.14.2

• Operating System and version: N/A (deployed on Kubernetes)

• Enabled features (icinga2 feature list):
  Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 journald livestatus opentsdb perfdata syslog mainlog
  Enabled features: api checker icingadb notification

• Icinga Web 2 version and modules (System - About):
  Icinga Web 2 - 2.12.1
  Loaded Modules
  icingadb - 1.1.3
  cube - 1.3.3
  director - 1.11.1
  incubator - 0.22.0
  reporting - 1.0.2
  x509 - 1.3.2

• Config validation (icinga2 daemon -C):
  [2024-12-03 09:57:51 +0000] information/cli: Icinga application loader (version: v2.14.2)
  [2024-12-03 09:57:51 +0000] information/cli: Loading configuration file(s).
  [2024-12-03 09:57:51 +0000] information/ConfigItem: Committing config item(s).
  [2024-12-03 09:57:51 +0000] information/ApiListener: My API identity: satellite-0
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 7 Downtimes.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 59 Users.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 2 TimePeriods.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1837 Services.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 162 Zones.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 5 NotificationCommands.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 2770 Notifications.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 236 Hosts.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 16 HostGroups.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 162 Endpoints.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1 ApiUser.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 1 ApiListener.
  [2024-12-03 09:57:52 +0000] information/ConfigItem: Instantiated 540 CheckCommands.
  [2024-12-03 09:57:52 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
  [2024-12-03 09:57:52 +0000] information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
  Here is the zones.conf from one of the satellites:
  object Endpoint "satellite-0" {
  // this is me
  }
  // the masters
  object Endpoint "master-0" {
  host = "master-0"
  port = "443"
  }
  // the masters
  object Endpoint "master-1" {
  host = "master-1"
  port = "443"
  }

// the other satellites
object Endpoint "satellite-1" {
host = "satellite-1"
port = "443"
}

object Zone "master" {
endpoints = [
"master-1",
"master-0"]
}

object Zone "satellite" {
endpoints = [
"satellite-1",
"satellite-0"]
parent = "master"
}

object Zone "global-templates" {
global = true
}

object Zone "director-global" {
global = true
}

Additional context

Not sure what other details to provide in this context, please advise.

Thanks!

Start by examining the notification and checker features and reproduce a Director configuration apply in the two-master, two-satellite setup. Use icinga2 daemon -C and the reported v2.14.2 configuration as the baseline; done means notifications are not sent while a host remains in a soft state and are sent only after it reaches a hard state.