Resolved notification sent with send_resolved set to false

What did you do?

During a flapping network event, HostMissing alerts (defined as 5min with 100% ping loss) fired. Because the alertmanager smtp smarthost was among those affected (indeed, is the precise HostMissing we're concerned with here), some notification deliveries failed, and that caused a NotificationsFailing alert as well.

I don't believe the inhibit_rules had any effect. I use HostMissing to suppress other alerts for an instance, but not itself.

My theory is that alertmanager attempted to send a "normal" firing notification but the delivery failed. It continued trying, and eventually succeeded. But it re-evaluated the template each time it re-attempted delivery. And, on successful re-attempt, the alert evaluated as Resolved. (This theory doesn't quite work out though, because ALERTS doesn't show the resolve until 13min after the [RESOLVED] notification.)

Timeline:

21:09:39: HostMissing goes pending
21:14:36: HostMissing fires
21:16:10: alertmanager_notifications_failed_total increments (first time)
21:16:16: NotificationsFailing fires [no pending state]
21:17: [RESOLVED] HostMissing notification delivered (timestamp from email)
21:17:15: alertmanager_notifications_failed_total increments (second time)
21:17:52: alertmanager_notifications_failed_total increments (third time)
21:22: [FIRING] NotificationsFailing notification delivered (timestamp from email)
21:24:50: alertmanager_notifications_failed_total increments (fourth time)
21:25:57: alertmanager_notifications_failed_total increments (fifth time)
21:26: [FIRING] HostMissing notification delivered (timestamp from email)
21:26:11: alertmanager_notifications_failed_total increments (sixth time)
21:26:56: alertmanager_notifications_failed_total increments (seventh time)
21:30:35: HostMissing resolves.

What did you expect to see?

First notification was the [FIRING] HostMissing alert. Possibly some repeat notifications during the event. And nothing else.

What did you see instead? Under which circumstances?

First notification was a [RESOLVED] HostMissing alert. Then, a [FIRING] NotificationsFailing alert about 5min later. And finally, a [FIRING] HostMissing alert came about 4min after that (9min after the original [RESOLVED] alert).

Environment

• System information:

# uname -srm
Linux 4.19.0-8-amd64 x86_64

• Alertmanager version:

# /usr/bin/prometheus-alertmanager --version
alertmanager, version 0.15.3+ds (branch: debian/sid, revision: 0.15.3+ds-3+b1)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20190608-09:56:55
  go version:       go1.11.6

• Prometheus version:

# /usr/bin/prometheus --version
prometheus, version 2.7.1+ds (branch: debian/sid, revision: 2.7.1+ds-3+b11)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20190608-09:54:04
  go version:       go1.11.6

• Alertmanager configuration file:

global:
   smtp_smarthost: 'mail.host:25'
   smtp_from: 'alertmanager@host'
   smtp_require_tls: false
   smtp_hello: host
templates:
 - '/etc/prometheus/alertmanager_templates/*.tmpl'
route:
   group_by: ['alertname', 'cluster', 'service']
   group_wait: 30s
   group_interval: 5m
   repeat_interval: 3h
   receiver: manager
   routes:
   # Suppress all severity=info "alerts"
   - receiver: blackhole
     match:
       severity: info
inhibit_rules:
 - source_match:
     alertname: 'HostMissing'
   equal: ['instance']
receivers:
 - name: 'manager'
   email_configs:
   - to: 'mygmail@gmail.com'
 - name: 'blackhole'
   # goes nowhere, on purpose

• Prometheus configuration file:

groups:
 - name: notifications
   rules:
   - alert: NotificationsFailing
     expr: floor(rate(alertmanager_notifications_failed_total[10m]) * 600) > 0
     annotations:
       summary: Notification Failures
       description: "{{ $value }} notification deliveries failed in the last 10 minutes."
- name: presence
   rules:
   - alert: HostMissing
     expr: ping_loss{presence!="ephemeral"} >= 100
     for: 5m
     annotations:
       summary: Host {{ $labels.instance }} unpingable

• Logs:

Note: I'm pretty sure these are truncated because these timestamps are later, but I couldn't find any others. The actual network flakiness lasted several more hours and sent other alerts, which may explain these logs.

# journalctl -u prometheus-alertmanager
-- Logs begin at Tue 2021-03-09 22:50:40 UTC, end at Wed 2021-03-10 02:48:33 UTC. --
Mar 09 23:27:09 myhost prometheus-alertmanager[4286]: level=error ts=2021-03-09T23:27:09.94889837Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="dial tcp [<smarthost_ip>]:25: connect: connection timed out"
Mar 09 23:27:09 myhost prometheus-alertmanager[4286]: level=error ts=2021-03-09T23:27:09.992604688Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="dial tcp [<smarthost_ip>]:25: connect: connection timed out"

No source files, tests, or entry points are named. Start by reproducing the notification retry sequence from the supplied Alertmanager and Prometheus configurations, then trace how failed deliveries retain or re-evaluate alert state. Done means the behavior is explained and a regression test covers the send_resolved=false case.