HA Alertmanager can send conflicting notifications in certain cases

When Alertmanager is configured for HA, peers are only made aware of a notification after it has been successfully sent to a receiver. In cases where the peers are running on a slightly offset timing and an alert is flapping, the peers can appear to "de-sync" and send conflicting notifications within seconds of each other.

E.g.

• Peer 0 restarts, and is now running some seconds behind peer 1 (in our case ~5s)
• Peer 1 starts sending notification for an aggregation group (in our case, the alert is firing)
• Peers all receive new alert state from the ruler for this same aggregation group (in our case, the alert resolves)
• Peer 0 checks notification log to dedup and sees no notifications, so starts sending its own after waiting <peer position> * peer_timeout, so 0 * peer_timeout
• Peer 1 finishes sending its notification and writes to the notification log (state: alerting)
• Peer 0 does the same (state: resolved)

A potential solution to this would be to gossip more detailed notification state (sending, retrying, failed, etc) in order to avoid multiple notifications being in-flight at the same time.

Alternatively, having all peers wait some non-zero multiple of peer_timeout could help mitigate this issue for long running notifications.

The issue names no files or tests; begin by tracing the HA notification flow, notification log, peer ordering, and peer_timeout behavior described here. Reproduce the flapping-alert timeline and define completion as preventing conflicting notifications from concurrent peers without losing the alerting or resolved notification.