`/-/ready` reports ready before cluster gossip has settled

What did you do?

While replacing Kubernetes nodes, we saw that active alerts in Alertmanager did not stick around when the new Alertmanager pods came up. We run 3 AM pods with a PodDisruptionBudget that allows 2 pods to go down, meaning one should always be ready. We also use a persistent volume for each instance. This is all set up using the Prometheus Operator. When replacing nodes, we used kubectl cordon and kubectl drain so that we can move the AM pods to new nodes in a way that respects the PodDisruptionBudget. This seems to indicate that the AM pods were marked as ready before the cluster was ready, despite us having a readiness probe using the /-/ready endpoint.

The impact of this is that we end up with orphaned alerts in Pagerduty, since Alertmanager doesn't know to send the resolve signal.

What did you expect to see?

I expected the active alerts to still exist after the AM pods are fully rotated.

I can see in the code that the readiness check doesn't actually check anything with regards to the cluster being ready, just the individual instance having an active HTTP listener.

https://github.com/prometheus/alertmanager/blob/main/ui/web.go#L79-L82

What did you see instead? Under which circumstances?

The new AM pods don't report any active alerts, and Pagerduty incidents must be resolved manually.

Environment

• Alertmanager version:

quay.io/prometheus/alertmanager:v0.21.0

• Prometheus version:

quay.io/prometheus/prometheus:v2.32.1

• Logs:

Here are logs showing AM pod startup. I deleted a pod and watch the old/new pod's status, and I captured the new pod's logs. It shows that the pod was marked ready at 18:09:36, but the logs say the gossip settled at 18:09:42.

Kubernetes readiness status. Before the jq error is the old pod, and after is the new pod.

{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:03:05Z","status":"True"}
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:30Z","status":"False"}
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:30Z","status":"False"}
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:30Z","status":"False"}
jq: error (at <stdin>:58843): Cannot iterate over null (null)
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:30Z","status":"False"}
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:30Z","status":"False"}
{"name":"alertmanager-alertmanager-cluster-0","lastTransitionTime":"2022-07-26T18:09:36Z","status":"True"}

Startup logs:

Defaulted container "alertmanager" out of: alertmanager, config-reloader
level=info ts=2022-07-26T18:09:32.431Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)"
level=info ts=2022-07-26T18:09:32.431Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"
level=info ts=2022-07-26T18:09:32.538Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2022-07-26T18:09:33.031Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2022-07-26T18:09:33.131Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2022-07-26T18:09:33.136Z caller=main.go:485 msg=Listening address=:9093
level=info ts=2022-07-26T18:09:34.538Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=3 elapsed=2.000114577s
level=info ts=2022-07-26T18:09:42.539Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.00075794s

Start with ui/web.go#L79-L82 and trace the readiness handler alongside the cluster startup path described in cluster.go. Compare readiness timing with the “gossip settled; proceeding” event in the supplied logs. Done means /-/ready does not report ready before cluster gossip has settled, while preserving the expected behavior for a fully initialized Alertmanager.