Satellite has problems with expired CRL

Describe the bug

A very simple configuration of an icinga2 satellite was unable to connect to our icinga2-master, because the CRL has expired. The daemon was not reloaded for 30 days, which is our maximum CRL age. This is basically the same issue we faced with #8501.

The main purpose of this satellite is to check master reachability and health.
We update our CRL every 6 hours and a restart/reload fixes the issue. So I think, that a running daemon is not correctly reloading the changed CRL. We only face this issue with this icinga2 instance! Other hosts are reloading correctly, but are running a more complex configuration.

Aug 06 08:39:39 icinga2-satellite icinga2[701]: API client disconnected for identity 'icinga2-master'
Aug 06 08:39:49 icinga2-satellite icinga2[701]: Certificate validation failed for endpoint 'icinga2-master': code 12: CRL has expired
Aug 06 08:39:59 icinga2-satellite icinga2[701]: Certificate validation failed for endpoint 'icinga2-master': code 12: CRL has expired
Aug 06 08:39:59 icinga2-satellite icinga2[701]: API client disconnected for identity 'icinga2-master'
Aug 06 08:39:59 icinga2-satellite icinga2[701]: API client disconnected for identity 'icinga2-master'
Aug 06 08:40:09 icinga2-satellite icinga2[701]: Certificate validation failed for endpoint 'icinga2-master': code 12: CRL has expired
Aug 06 08:40:19 icinga2-satellite icinga2[701]: API client disconnected for identity 'icinga2-master'
Aug 06 08:40:19 icinga2-satellite icinga2[701]: Certificate validation failed for endpoint 'icinga2-master': code 12: CRL has expired
Aug 06 08:40:19 icinga2-satellite icinga2[701]: API client disconnected for identity 'icinga2-master'
Aug 06 08:40:29 icinga2-satellite icinga2[701]: Certificate validation failed for endpoint 'icinga2-master': code 12: CRL has expired

To Reproduce

Start icinga2, wait until CRL expiration and connections start to fail if the master drops the connection, because of config reloads.

# zones.conf
object Endpoint "icinga2-master" {
        host = "icinga2-master"
        port = "5665"
}
object Zone "master" {
        endpoints = [ "icinga2-master" ]
}
object Endpoint "icinga2-satellite" {

}

object Zone "icinga2-satellite" {
        endpoints = [ "icinga2-satellite" ]
        parent = "master"
}

Expected behavior

The daemon periodically reloads the CRL or monitors the CRL for changes.

Your Environment

icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.4-1)

System information:
Platform: Debian GNU/Linux
Platform version: 10 (buster)
Kernel: Linux
Kernel version: 4.19.0-21-amd64
Architecture: x86_64

Enabled features: api checker command mainlog notification syslog

Config validation:

[2022-08-15 14:48:59 +0200] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-08-15 14:48:59 +0200] information/cli: Loading configuration file(s).
[2022-08-15 14:48:59 +0200] information/ConfigItem: Committing config item(s).
[2022-08-15 14:48:59 +0200] information/ApiListener: My API identity: icinga2-satellite
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 2 Notifications.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 Host.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 EventCommand.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 2 Zones.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 2 Endpoints.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 159 CheckCommands.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 9 UserGroups.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 7 TimePeriods.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 28 Users.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 1 Service.
[2022-08-15 14:48:59 +0200] information/ConfigItem: Instantiated 7 NotificationCommands.
[2022-08-15 14:48:59 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-08-15 14:48:59 +0200] information/cli: Finished validating the configuration file(s).

Additional context

We are rolling out our own certificate infrastructure and are not relying on icinga2 pki.

Start with the API connection and certificate-validation path used by the satellite, then trace how the CRL is loaded and refreshed while icinga2 is running. Use the minimal zones.conf configuration and the reported expired-CRL scenario to verify that updated CRLs are recognized without restarting or reloading the daemon.