prometheus / prometheus/alertmanager

Muting interacts with group_interval in unexpected (?) fashion

Open
#2,496 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/notify component/silences kind/change
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

If all alerts in a notification group are muted (silenced or inhibited) at the time the group would notify for the first time, no notification is sent. So far so good. However, if an alert in that group becomes unmuted, the group won't notify immediately. The muted notification counts as "this group has already notified", so that the next notification will only be sent once group_interval has passed. This is particularly problematic with a very long group_interval, let's say one day. In extreme cases, this might lead to the group to never notify (if at the time the group_interval runs out another condition is met that mutes the alert at this time again, particularly easy with the new time-based muting feature).

More discussion here.

I can totally imagine that the current semantics has its reasons, and that changing it is not only difficult to implement but might create some other unexpected side effects. I still want to file this issue so that we are aware of this behavior. Should the conclusion be that it cannot be changed, we can declare it a feature, document it properly, and close this as "works as intended". Hopefully, though, we can find a better way.

Note that the same thought process needed to resolve this issue might also help with the long standing problem of when to send a resolve notification in case an alert is muted.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the reported behavior and the linked discussion about muted alerts, group_interval, and resolve notifications. Trace how Alertmanager decides whether a group has already notified and compare that with the intended muting semantics. Done means reaching an agreed behavior and either implementing it with coverage or documenting that the current behavior is intentional.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.