Alertmanager cluster peers have spiky routines

We have observed that the Alertmanager cluster peer on position 0 (which is the first to send notifications) has almost flat routines metric.
While the other 2 peers in following positions have more spiky routines up to 3K routines (+30%).
In the below graph we can see how the yellow instance at position zero had a flat routines graph but when the positions are swapped by renaming peers, the blue is at position zero. The switch in positions moves the spiky routines symptom between the peers.

So the fanout stage creates all these go routines here https://github.com/prometheus/alertmanager/blob/edb9a4d93637d500cb11916a1d3dfa4e41c5a4fa/notify/notify.go#L491-L514

and then the wait stage will wait peer timeout * position here https://github.com/prometheus/alertmanager/blob/edb9a4d93637d500cb11916a1d3dfa4e41c5a4fa/notify/notify.go#L599-L607

This seems very unoptimised and wasteful, we could potentially redesign these stages to wait before creating the routines instead of creating, waiting and then destroying.

Start with notify/notify.go at the fanout stage around lines 491-514 and the wait stage around lines 599-607. Trace how peer position affects routine creation and waiting, then determine how the stages can wait before creating routines. Done means the peer-position-dependent routine spikes are eliminated without changing notification behavior.