prometheus / prometheus/alertmanager

Have a way to mute alert until it's resolved to receive a resolved notification once it's fixed

Open
#3,825 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

Let's say I have an outage on one of my server I'm monitoring and it's inaccessible but I don't know how long it's gonna take to fix it, so I'm muting it for a really long time.

With this approach, I won't receive any resolved notifications, so to check if the alert is fixed I need to go to my alerts list to see if it's still firing, and given that I've muted it for a long time I also need to remove the mute to know if it's firing again.

What would be nice to have:

  • I have a server outage
  • an alert is triggered
  • I create a new mute and somehow specify that I need it to be active until the alert is resolved
  • a server fixes itself
  • a resolved alert notification is dispatched
  • a mute is removed
  • if a server starts misbehaving again and a new alert is triggered, I'm receiving an alert notification again

Pretty sure this would have a lot of cases that'll make it difficult, like if a mute has a lot of active alerts, but still would be really awesome to have.

Do you guys think it's manageable?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing Alertmanager's mute lifecycle and how resolved notifications are dispatched. Define how a mute should end when its alert resolves, including behavior for multiple active alerts, then identify the relevant implementation and tests before proposing a design.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.