Record more info about alert to know it's triggered by error or real data and when

Issue by ylwu-amzn
Friday May 08, 2020 at 03:14 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/201

Is your feature request related to a problem? Please describe.

Currently monitor will create an alert if any error happen. But we can't know the historical alert triggered by error or real data as the state becomes "COMPLETED" or "ACKNOWLEDGED". And if alert triggered by some error first, then triggered by real data later, we don't know when the real data alert started. Example:

1. Alert triggered by timeout error at 5:00 PM .
2. Monitor next run at 5:05PM, timeout error gone, but monitor's input meets trigger condition, alert will stay at active.
3. Monitor run at 5:10PM, no error, and monitor input doesn't meet trigger condition, alert becomes COMPLETED.

In this example, we can't know the real data alert starts from 5:05PM.

Describe the solution you'd like

• Option1:
  Suggestion from @dbbaughe

Add a counter map on the alert to keep track of each states number for the lifetime of an alert
So when it’s completed you end up with a nice map of
Alert: { error: 1, active: 9, acknowledged: 5 }
i.e. the alert during the 15 executions was in error state once, active 9 times, and acknowledged 5

Question:
Can we add two more fields, so we can query easily?

1. error_start_time
2. real_data_alert_start_time
   Or something else is ok, only if we can query easily to get which alert triggered by real data and know the real data alert start time.

• Option2:

Add alert type filed. For alert triggered by error, we can call it "ERROR_ALERT". For real data alert, we can call "REAL_ALERT". For the above example:

1. Alert triggered by timeout error at 5:00 PM . We create a "ERROR_ALERT".
2. Monitor next run at 5:05PM, timeout error gone, but monitor's input meets trigger condition, alert will stay at active. We close the "ERROR_ALERT" and create a "REAL_ALERT".
3. Monitor run at 5:10PM, no error, and monitor input doesn't meet trigger condition, alert becomes COMPLETED. We close the "REAL_ALERT"

We can show different style for different alert types on Kibana page and surface error message for "ERROR_ALERT".

Start by reviewing the monitor and alert lifecycle described in the issue, including error-triggered alerts, real-data alerts, and transitions to COMPLETED or ACKNOWLEDGED. Compare the two proposed designs and determine how alert history, alert type, and start times should be queried; the issue does not name files or tests, so the implementation entry point is not specified.