prometheus / prometheus/alertmanager
[Feature] Additional validation to prevent extremely broad silences
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
A silence was created with such a broad set of matchers that many alerts were silenced unintentionally.
What did you expect to see?
When a silence is created, it is validated against some additional criteria defined by the administrator (e.g. number of matchers).
Proposal/More context
We have a setup where engineers can create silences in Alertmanager using any combination of matchers. This lead to an instance recently where an engineer unknowingly muted alerts for multiple services by creating a silence with only a single matcher. This matcher was so broad that it impacted the delivery of multiple other alerts.
In addition to the current validation being done when a new silence is created it might be worth defining some additional parameters that can be used to assess the “validity” of a silence. The values of which can be defined by whoever is deploying Alertmanager, e.g.:
min_matchers: 3 # silence requires at least 3 matchers
denylist: # silence not valid if only the following matchers are used
- severity
- prometheus
Can totally understand if this is out of scope of Alertmanager but just wanted to raise in case this is something that’s been considered before. I don’t think there really is a perfect solution to this but would be good to get any thoughts on how this could potentially be managed without just restricting access to the Alertmanager UI. The Preview Alerts feature is very useful so maybe that’s a good enough solution for the timebeing.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by reviewing the existing validation performed when a silence is created and the Preview Alerts feature, then determine how administrator-defined criteria such as minimum matchers or denylisted matcher sets could be specified and what validation behavior would count as complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100