prometheus / prometheus/alertmanager
Feature Request - Temporary Suspension of Alert Resolution for noisy and flapping alert
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
Say an alert is having for value of 5 mins.
Let's assume a scenario where this alert triggers every 20 mins once and closes within 10 mins.
We will have 100+ alerts in a day in this scenario.
The alert may be misbehaving for 1% of targets but that 1% contributes to 99% of noise for operations.
Of course, we can use increase or increases_over_time to solve this problem.
But this might water down the alert for the rest of the 99% targets.
At present, an alert is active while its expression is true. Can alertmanager solve this problem of noisy notifications by tracking the number of times an alert went to a firing state in X amount of time?
This is not flap detection. Instead, a new feature which will keep the alert open for an extended period as it's too noisy.
The idea here is to keep the open alert in a firing state.
The alert should not auto-resolve until the suspension period is over. That way the alert will be open for a longer period instead of closing and reopening frequently.
This should also reduce overall CPU/memory use.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; start by tracing Alertmanager's alert firing and resolution lifecycle, including how notification state is tracked. Define the suspension-period behavior for repeatedly firing alerts, then add coverage showing that an alert remains open until the suspension period ends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100