Icinga Flapping Detection prevents “resolved” notifications from being sent

Describe the bug

As we know the flapping detection mechanism in Icinga2 suppress notifications. And there was a feature/fix introduced in Icinga 2.13.3 – that if there is flapping period for any service in Icinga2 and if there is a state change (consider the states just before and after the flapping starts and ends for comparison) for the service then a notification will be sent. We will observed the following two cases:

1. If there is a state change from OK to CRITICAL, a notification is being sent just after the flapping ends.
2. If there is a state change from CRITICAL to OK, no notification is being sent just after the flapping ends.

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this issue. Include configuration, logs, etc. to reproduce, if relevant.

1. I created a service which will monitor a deployment/pod in kubernetes system. Prometheus query was used to check the pod - kube_pod_status_ready{ condition="false",pod=~"(test).*"}
2. I checked the service in icinga it was in OK state.
3. I changed the deployment definition (json file) pointing to wrong image and re-deployed it. It produced an ImagePullOff error.
4. After some time the Icinga service was in CRITICAL state.
5. I changed the deployment definition (json file) pointing to correct image and re-deployed it. Deployment/pod was in ready/running state.
6. After some time the Icinga service was in OK state.
7. Repeated the above steps - 3 to 6 to create flapping.
8. After this just monitored the icinga - if it state before flapping was in CRITICAL and the flapping occured and then the service become in OK state and then flapping turned off.
   9 waited for the notification being sent. But the notification for OK status was not sent.

Expected behavior

The desired behavior is to sent notification in the 2nd case as well which was described in the section - "Describe the bug".

Illustration

Please find the below list of events happening in Icinga2 to understand the problem statement

1. An alert triggers
2. Notification for CRITICAL state triggered
3. Flapping turns ON
4. Alert resolved
5. Flapping turns OFF
6. Notification for OK State triggered <------- This doesn't happen
   Our expectation was that at step 6, a notification must be triggered for the state change (from CRITICAL to OK), but not happening.
   Please find the same scenario with time stamps :-
7. [11:03:20] An alert triggers
8. [11:03:20] Notification for CRITICAL state triggered to Pagerduty
9. [11:03:02] Flapping turns ON
10. [11:03:02] Alert resolved
11. [11:12:57] Flapping turns OFF
12. [] Notification(Pagerduty notification) for OK State triggered <------- This doesn't happen
    And the same illustrated via screen shot with increasing order of time from bottom to up.

Screenshots

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Icinga Web 2 version and modules (System - About): 2.9.5

• Web browser used: Version 103.0.5060.114 (Official Build) (64-bit)

• Icinga 2 version used (icinga2 --version):
  icinga2-common-2.13.4-1.el7.icinga.x86_64
  icinga2-bin-2.13.4-1.el7.icinga.x86_64
  icinga2-2.13.4-1.el7.icinga.x86_64
  icinga2-ido-mysql-2.13.4-1.el7.icinga.x86_64
  icinga2-selinux-2.13.4-1.el7.icinga.x86_64

• PHP version used (php --version):
  rh-php73-php-common-7.3.29-1.el7.x86_64
  php-cli-7.1.33-19.el7.remi.x86_64
  rh-php73-php-gd-7.3.29-1.el7.x86_64
  rh-php73-php-soap-7.3.29-1.el7.x86_64
  php-7.1.33-19.el7.remi.x86_64
  php-gd-7.1.33-19.el7.remi.x86_64
  rh-php71-runtime-1-1.el7.x86_64
  rh-php71-php-pdo-7.1.30-2.el7.x86_64
  rh-php71-php-mysqlnd-7.1.30-2.el7.x86_64
  rh-php73-php-json-7.3.29-1.el7.x86_64
  php-json-7.1.33-19.el7.remi.x86_64
  rh-php73-php-pdo-7.3.29-1.el7.x86_64
  rh-php73-php-intl-7.3.29-1.el7.x86_64
  rh-php73-php-ldap-7.3.29-1.el7.x86_64
  rh-php73-php-pgsql-7.3.29-1.el7.x86_64
  php-pdo-7.1.33-19.el7.remi.x86_64
  rh-php73-php-xml-7.3.29-1.el7.x86_64
  rh-php73-php-fpm-7.3.29-1.el7.x86_64
  rh-php73-php-cli-7.3.29-1.el7.x86_64
  rh-php73-php-mbstring-7.3.29-1.el7.x86_64
  icinga-php-thirdparty-0.10.0-2.el7.icinga.noarch
  php-mcrypt-7.1.33-19.el7.remi.x86_64
  php-opcache-7.1.33-19.el7.remi.x86_64
  icinga-php-common-1.0.0-1.el7.icinga.noarch
  rh-php71-php-zip-7.1.30-2.el7.x86_64
  rh-php71-php-common-7.1.30-2.el7.x86_64
  rh-php71-php-fpm-7.1.30-2.el7.x86_64
  rh-php71-php-intl-7.1.30-2.el7.x86_64
  rh-php71-php-xml-7.1.30-2.el7.x86_64
  rh-php73-php-zip-7.3.29-1.el7.x86_64
  php-common-7.1.33-19.el7.remi.x86_64
  icinga-php-library-0.7.0-1.el7.icinga.noarch
  rh-php73-php-mysqlnd-7.3.29-1.el7.x86_64
  icingaweb2-vendor-lessphp-2.9.6-1.el7.icinga.noarch
  php-Icinga-2.9.6-1.el7.icinga.noarch
  php-mysqlnd-7.1.33-19.el7.remi.x86_64
  php-xml-7.1.33-19.el7.remi.x86_64
  rh-php73-runtime-1-1.el7.x86_64
  rh-php71-php-json-7.1.30-2.el7.x86_64

• Server operating system and version: CentOS Linux release 7.9.2009

Additional context

No source file or test is identified in the report. Start by reproducing the described Icinga2 flapping sequence with the Prometheus-backed service and trace the notification flow; done means an OK notification is sent when flapping ends after a CRITICAL-to-OK transition.