Aggregation Groups result in too many go routines

Aggregation groups run in dedicated go routines, this means that alertmanager can create thousands of routines, depending on the number of unique alerts it receives and grouping config: https://github.com/prometheus/alertmanager/blob/f6b942cf9b3a503d59192eada300d2ad97cba82f/dispatch/dispatch.go#L350

These routines are usually waiting in a select statement for the next ticker which seems very wasteful: https://github.com/prometheus/alertmanager/blob/f6b942cf9b3a503d59192eada300d2ad97cba82f/dispatch/dispatch.go#L440-L442

Some graphs from our deployment:

A better approach could be creating one go routine per receiver instead as aggregation groups share receivers. This should reduce the number of go routines significantly.
Aggregation groups can depend on receiver based trigger to flush notifications.

Aggregation groups also support a global limit: https://github.com/prometheus/alertmanager/blob/f6b942cf9b3a503d59192eada300d2ad97cba82f/dispatch/dispatch.go#L333-L338
but this is currently not used(nil limiter): https://github.com/prometheus/alertmanager/blob/f6b942cf9b3a503d59192eada300d2ad97cba82f/cmd/alertmanager/main.go#L495

I think to protect alertmanager under load, we should implement a per receiver limiter.
If alerts sent to a receiver are not acted upon it can be assumed that alert manager can limit the number of aggregation groups created for that receiver.

Start in dispatch/dispatch.go at the aggregation-group goroutine and global limiter sections linked in the issue, then inspect cmd/alertmanager/main.go around line 495 where the limiter is nil. Determine how receiver-based triggering and limiting should fit the existing dispatch flow. Done means reducing per-group goroutines and wiring a per-receiver limiter to protect Alertmanager under load.