Ability to configure notification when alert auto heals

Issue by alfredolopezzz
Wednesday Dec 23, 2020 at 12:23 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/330

Is your feature request related to a problem? Please describe.
We have to manually ACK and close ALL alerts that ODFE-alerting sends to us (we are using Opsgenie for routing alerts to on-call engineers) even if our systems manage to auto-heal correctly (ie, metrics go back to normal). This is not scalable when managing a system with lots of alerts.

Describe the solution you'd like
We would like to be able to receive notifications when a monitor "Trigger" transitions its status from "alert triggered" to "no alerts". Right now when creating a Trigger, in the "Configure actions" section, it is only possible to specify one single message which will be used to send a notification to the destination when trigger enters the alert status.

Describe alternatives you've considered
This is a showstopper for us, we are considering moving our observability platform out of ODFE.

Start by tracing the monitor Trigger and its Configure actions flow, then inspect how transitions from alert triggered to no alerts are handled. Done means a configurable notification can be sent when an alert auto-heals, alongside the existing notification for entering alert status.