Alert housekeeping and Pagerduty

What did you do?

We're using Alertmanager pushing alerts into Pagerduty at rather large scale. We're often applying silences in an automated fashion (eg. during some rollouts), but also manually sometimes.

What did you expect to see?

A list of current, non-silenced alerts in Pagerduty, which is the primary overview on open issues for our on-call engineer.

This especially also applies to cases involving silences and (short) Alertmanager service disruptions.

What did you see instead? Under which circumstances?

Alerts get pushed to Pagerduty as expected. Things start to break as soon as

• silences are removed while alerts exist (related to #226) or
• an alert ceases to exist in Prometheus while Alertmanager does not work for some reason

In either case, we will have stuck alerts in Pagerduty. It is hard to decide whether an alert is actually an already-silenced issue or already "acknowledged" in Alertmanager by adding a silence.

We were able to tackle the Alertmanager disruption case by running a HA cluster of Alertmanager, but the silence issue is really annoying.

From our understanding, there are actually two underlying issues:

1. Similar to #226, Alertmanager supresses resolved notifications (to Pagerduty and others) if a silence exists. A silence should "mute alerts" for a given time, which might require different behavior for two different use cases:

   • Sending to something like a chat channel should probably be suppressed (here, the resolved message would probably just disturb when the silence exists, nobody is tracking state of alerts) and
   • Sending alerts to an alerting system tracking state of existing alerts, where we definitely want all alerts to be resolved.
     I'd even suppose a "ternary" configuration which allows generally supressing resolved messages during silences (chat targets), not supressing resolved messages during silences (Pagerduty, ...) and the option to artificially resolve alerts when silences are created and send new alerts when the silences ends -- this would actually represent a real "sync" use case with systems like Pagerduty.

   Maybe extend send_resolved for this?

   Getting this handled would be very helpful: This is important in situations we usually do have some kind of issue, and want to block out some kinds of alerts for the moment that we will not (be able to) handle for the moment.

2. This is worse to implement and probably rather complicated, especially in HA use cases. Alertmanager did not work for some time, so it has no record of what happened. To resolve this issue, Alertmanager would have to scrape the alert target's state (ie. Pagerduty API) and resolve Alertmanager-created alerts that do not exist any more. Or track state of pushed alerts and compare.

   As this happens very rarely, we might be able to handle this by runing Pagerduty's auto-resolve setting together with Pagerduty's repeat_interval to rather low values (maybe 2h vs 1h), to automatically cleanup Pagerduty. We have not tested this yet.

I think Alertmanager's use case should either be refined/restricted (so both or one of the individual issues can be closed as "won't do"), or broaden the definition and consider such an implementation useful for Alertmanager. We'd probably love to help out with some code if we potentially can get it upstream (but might take some time until we prioritize this), or we have to rethink our alerting strategy.

Environment

• System information:

not relevant

• Alertmanager version:

0.15.3

• Prometheus version:

2.7.1

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
  http_config:
    proxy_url: http://<snip>
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
route:
  receiver: warning
  group_by:
  - tenant
  routes:
  - receiver: critical
    match:
      severity: critical
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1d
receivers:
- name: critical
  pagerduty_configs:
  - send_resolved: true
    http_config:
      proxy_url: http://<snip>
    routing_key: <secret>
    url: https://events.pagerduty.com/v2/enqueue
    client: '{{ template "pagerduty.default.client" . }}'
    client_url: '{{ template "pagerduty.default.clientURL" . }}'
<lots of templating stuff omitted>
<further receivers omitted>

• Prometheus configuration file:

Not relevant?

• Logs:

None considered relevant?

Start by reviewing Alertmanager's current silence handling, resolved-notification behavior, and the PagerDuty configuration shown in the issue, including send_resolved and repeat_interval. Compare the two proposed cases with related issue #226 and determine whether the desired scope is silence synchronization, recovery from Alertmanager outages, or a narrower clarification. Done requires an agreed behavior and implementation scope for PagerDuty and other notification targets.