Fail to send alerts to alertmanager due to EOF

What did you do?

We configured multiple(1000+) prometheus instance to send alerts to one alertmanager, at the beginning, everything is fine, after a few minutes, we found some prometheus failed to send alerts due to the error:

level=error ts=2021-07-01T12:33:16.623Z caller=notifier.go:527 component=notifier alertmanager=https://alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="Post \"https://alertmanager/api/v2/alerts\": EOF"

What did you expect to see?

All prometheus instances send alerts to Alertmanager successfully.

What did you see instead? Under which circumstances?

We found some prometheus failed to send alerts due to the error:

level=error ts=2021-07-01T12:33:16.623Z caller=notifier.go:527 component=notifier alertmanager=https://alertmanager/api/v2/alerts count=1 msg="Error sending alert" err="Post \"https://alertmanager/api/v2/alerts\": EOF"

Environment

Prometheus v2.4.2
Alertmanager 0.21.0

Checked the source code for the client http transport settings, we found that each prometheus keeps large max idle connection(20000 in total and 1000 per host) to send alerts to Alertmanager.

See: https://github.com/prometheus/common/blob/a1b6ede20323252d2b99a0f57178a4b7d364d0ca/config/http_config.go#L370-L380

I wonder why we hard code these settings, they should be customized by end users. For my case, we have 1000+ Prometheus to send alerts to one Alertmanager, the client connection numbers will be *1000 for the upstream Alertmanager.

If there are some rate limit in upstream, the connection will be closed. In any case, we need to make sure the connection from each Prometheus instance can be customized.

Start with notifier.go around line 527 and the referenced common/config/http_config.go transport settings. Trace how idle connection limits are created for Alertmanager requests and review the issue discussion about customizing them. Done means the relevant connection settings can be customized per Prometheus instance and the existing alert-sending behavior remains covered by tests.