prometheus / prometheus/alertmanager
Muting interacts with group_interval in unexpected (?) fashion
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
If all alerts in a notification group are muted (silenced or inhibited) at the time the group would notify for the first time, no notification is sent. So far so good. However, if an alert in that group becomes unmuted, the group won't notify immediately. The muted notification counts as "this group has already notified", so that the next notification will only be sent once group_interval has passed. This is particularly problematic with a very long group_interval, let's say one day. In extreme cases, this might lead to the group to never notify (if at the time the group_interval runs out another condition is met that mutes the alert at this time again, particularly easy with the new time-based muting feature).
More discussion here.
I can totally imagine that the current semantics has its reasons, and that changing it is not only difficult to implement but might create some other unexpected side effects. I still want to file this issue so that we are aware of this behavior. Should the conclusion be that it cannot be changed, we can declare it a feature, document it properly, and close this as "works as intended". Hopefully, though, we can find a better way.
Note that the same thought process needed to resolve this issue might also help with the long standing problem of when to send a resolve notification in case an alert is muted.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the reported behavior and the linked discussion about muted alerts, group_interval, and resolve notifications. Trace how Alertmanager decides whether a group has already notified and compare that with the intended muting semantics. Done means reaching an agreed behavior and either implementing it with coverage or documenting that the current behavior is intentional.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100