prometheus / prometheus/alertmanager
Have a way to mute alert until it's resolved to receive a resolved notification once it's fixed
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
Let's say I have an outage on one of my server I'm monitoring and it's inaccessible but I don't know how long it's gonna take to fix it, so I'm muting it for a really long time.
With this approach, I won't receive any resolved notifications, so to check if the alert is fixed I need to go to my alerts list to see if it's still firing, and given that I've muted it for a long time I also need to remove the mute to know if it's firing again.
What would be nice to have:
- I have a server outage
- an alert is triggered
- I create a new mute and somehow specify that I need it to be active until the alert is resolved
- a server fixes itself
- a resolved alert notification is dispatched
- a mute is removed
- if a server starts misbehaving again and a new alert is triggered, I'm receiving an alert notification again
Pretty sure this would have a lot of cases that'll make it difficult, like if a mute has a lot of active alerts, but still would be really awesome to have.
Do you guys think it's manageable?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Alertmanager's mute lifecycle and how resolved notifications are dispatched. Define how a mute should end when its alert resolves, including behavior for multiple active alerts, then identify the relevant implementation and tests before proposing a design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100