Icinga 2.14.0 not longer synchronizes modified attributes to satellites (regression from 2.13.7)

Describe the bug

Since we upgraded to 2.14.0 we experince issues with our satellites where "old" values reappear. It seems as all changes in modified-attributes.conf are no longer synchronized to the satellites. Services on satellite use values as specified in original_attributes in IcingaWeb2 (even though the correct ones are shown in the UI). As a consequence checks use incorrect/old vars.

This happens with existing/upgraded satellites. It also occurs on new satellites. We create all our services via API so this happens quite frequently for us and very reproducibly.

We verified that all files in /var/lib/icinga2/api are the same on master and satellite. Files in /var/lib/icinga2/api do not contain any modified attributes (neither on master nor on the satellites). Modified attributes can be found in /var/lib/icinga2/modified-attributes.conf on the master only.

This worked fine in 2.13.7 (did not try 2.13.8 yet). In 2.13.7 modified attributes would be merged into the files in /var/lib/icinga2/api on the satellite.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

1. Install master
2. Add service via API
3. Change service via API (i.e. change vars)
4. Add satellite
5. Observe check execution

Satellite will use the vars from step (2) but not from step (3).

Expected behavior

Satellite will use changed vars from step (3).

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version):

$ icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.14.0)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 11 (bullseye)
  Kernel: Linux
  Kernel version: 6.2.0-1011-aws
  Architecture: x86_64

Build information:
  Compiler: GNU 10.2.1
  Build host: buildkitsandbox
  OpenSSL version: OpenSSL 1.1.1n  15 Mar 2022

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

• Operating System and version: Debian bullseye (from official docker image - see above)
• Enabled features (icinga2 feature list):
  • Satellites: Enabled features: api checker notification syslog
  • Master: Enabled features: api checker icingadb notification syslog
• Icinga Web 2 version and modules (System - About):
• Config validation (icinga2 daemon -C):
  On master:

$ icinga2 daemon -C
[2023-09-27 20:25:58 +0000] information/cli: Icinga application loader (version: v2.14.0)
[2023-09-27 20:25:58 +0000] information/cli: Loading configuration file(s).
[2023-09-27 20:26:00 +0000] information/ConfigItem: Committing config item(s).
[2023-09-27 20:26:00 +0000] information/ApiListener: My API identity: xxx
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 SyslogLogger.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 6762 Dependencies.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 35 Comments.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 71 Users.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 405 ServiceGroups.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 8214 Services.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 14 Zones.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 8215 Notifications.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 12 Hosts.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 12 Endpoints.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1771 Downtimes.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 16 ApiUsers.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 263 CheckCommands.
[2023-09-27 20:26:01 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-09-27 20:26:01 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-09-27 20:26:01 +0000] information/cli: Finished validating the configuration file(s).

On a fresh satellite y:

$ icinga2 daemon -C
[2023-09-27 20:23:57 +0000] information/cli: Icinga application loader (version: v2.14.0)
[2023-09-27 20:23:57 +0000] information/cli: Loading configuration file(s).
[2023-09-27 20:23:57 +0000] information/ConfigItem: Committing config item(s).
[2023-09-27 20:23:57 +0000] information/ApiListener: My API identity: yyy
[2023-09-27 20:23:58 +0000] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 SyslogLogger.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1641 Dependencies.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 35 Users.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 112 ServiceGroups.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1757 Services.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 4 Zones.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 3511 Notifications.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 Host.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 476 Downtimes.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-09-27 20:23:58 +0000] information/ConfigItem: Instantiated 263 CheckCommands.
[2023-09-27 20:23:58 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-09-27 20:23:58 +0000] information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

$ icinga2 object list --type Endpoint

Object 'xxx of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 11:1-11:61
  * __name = "xxx"
  * host = "xxx"
    % = modified in '/etc/icinga2/zones.conf', lines 12:3-12:54
  * log_duration = 86400
  * name = "xxx"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/zones.conf', lines 13:3-13:15
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 61
    * last_line = 11
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "xxx" ]
    % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:61
  * type = "Endpoint"
  * zone = "master"
    % = modified in '/etc/icinga2/zones.conf', lines 14:3-14:17

Object 'yyy' of type 'Endpoint':
  % declared in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 5:1-5:45
  * __name = "yyy"
  * host = "yyy"
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 6:3-6:38
  * log_duration = 86400
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 8:3-8:19
  * name = "yyy"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 7:3-7:13
  * source_location
    * first_column = 1
    * first_line = 5
    * last_column = 45
    * last_line = 5
    * path = "/etc/icinga2/conf.d/satellite-config/y.conf"
  * templates = [ "yyy" ]
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 5:1-5:45
  * type = "Endpoint"
  * zone = ""

$ icinga2 object list --type Zone

Object 'master' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 7:1-7:20
  * __name = "master"
  * endpoints = [ "xxx" ]
    % = modified in '/etc/icinga2/zones.conf', lines 8:3-8:63
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 7
    * last_column = 20
    * last_line = 7
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 7:1-7:20
  * type = "Zone"
  * zone = ""

Object 'y-satellite' of type 'Zone':
  % declared in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 1:0-1:28
  * __name = "y-satellite"
  * endpoints = [ "yyy" ]
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 2:3-2:47
  * global = false
  * name = "y-satellite"
  * package = "_etc"
  * parent = "master"
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 3:3-3:19
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 28
    * last_line = 1
    * path = "/etc/icinga2/conf.d/satellite-config/y.conf"
  * templates = [ "y-satellite" ]
    % = modified in '/etc/icinga2/conf.d/satellite-config/y.conf', lines 1:0-1:28
  * type = "Zone"
  * zone = ""

Additional context

We run Icinga on top of Kubernetes with an operator to create checks/services. Setup is fully automated so nothing has been tampered manually.

After downgrading back to 2.13.7 we found this:

• /var/lib/icinga2/modified-attributes.conf contains 0 bytes on the satellite
• API changes are directly reflected in synchronized filed
• I could not find anything in the changelog which indicates a changed synchronization behaviour
• Starting satellites and master is much faster in 2.14.0 compared to 2.13.7 (30s vs 2 minutes)

Begin with the API synchronization path and handling of /var/lib/icinga2/modified-attributes.conf, comparing 2.14.0 with 2.13.7. Reproduce the listed master/API and new-satellite steps, checking /var/lib/icinga2/api and the satellite's check vars. Done means the changed vars from step (3) are synchronized and used by the satellite.