prometheus / prometheus/alertmanager
Expose which webhook endpoint is failing in AlertmanagerClusterFailedToSendAlerts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
We have multiple webhook configurations for a receiver. If one of these webhook endpoint is failing so we can't send alerts to them, the AlertmanagerClusterFailedToSendAlerts triggers, but it doesn't say which webhook is affected.
It would be great if the metric "alertmanager_notifications_failed_total" would include a label which is the endpoint url itself, so I could include it in the message.
What did you do?
Configured multiple webhooks and tried to discover which one is failing exactly.
What did you expect to see?
The exact webhook endpoint url, which is failing to send alerts to.
What did you see instead? Under which circumstances?
Only the ingeration label is exposed, which has the value "webhook".
- Alertmanager version:
0.19.0
- Prometheus version:
2.13.1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the alertmanager_notifications_failed_total metric and the AlertmanagerClusterFailedToSendAlerts alert through the webhook notification handling. Check how multiple webhook endpoints are represented and how metric labels are defined; done means the failing endpoint can be distinguished in the resulting alert message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100