prometheus / prometheus/alertmanager

Feature Request - Temporary Suspension of Alert Resolution for noisy and flapping alert

Open
#3,048 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

Say an alert is having for value of 5 mins.

Let's assume a scenario where this alert triggers every 20 mins once and closes within 10 mins.
We will have 100+ alerts in a day in this scenario.
The alert may be misbehaving for 1% of targets but that 1% contributes to 99% of noise for operations.

Of course, we can use increase or increases_over_time to solve this problem.
But this might water down the alert for the rest of the 99% targets.
At present, an alert is active while its expression is true. Can alertmanager solve this problem of noisy notifications by tracking the number of times an alert went to a firing state in X amount of time?

This is not flap detection. Instead, a new feature which will keep the alert open for an extended period as it's too noisy.

The idea here is to keep the open alert in a firing state.
The alert should not auto-resolve until the suspension period is over. That way the alert will be open for a longer period instead of closing and reopening frequently.
This should also reduce overall CPU/memory use.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests; start by tracing Alertmanager's alert firing and resolution lifecycle, including how notification state is tracked. Define the suspension-period behavior for repeatedly firing alerts, then add coverage showing that an alert remains open until the suspension period ends.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.