prometheus / prometheus/alertmanager

Identify if an firing alert is repeated

Open
#2,921 1 comment 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
Trying to identify a repeated alert in a message template
What did you expect to see?
Repeated: Alert_title
What did you see instead? Under which circumstances?
Alert_title
Environment
Linux

  • System information:
    Linux 5.4.0-107-generic
  • Alertmanager version:
    alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)
    build user: root@265f14f5c6fc
    build date: 20220325-09:31:33
    go version: go1.17.8
    platform: linux/amd64
  • Prometheus version:
    prometheus, version 2.34.0 (branch: HEAD, revision: 881111fec4332c33094a6fb2680c71fffc427275)
    build user: root@121ad7ea5487
    build date: 20220315-15:18:00
    go version: go1.17.8
    platform: linux/amd64

I'm trying to find a way to tell if a firing alert is a repeated one. Why I want this is because it would be nice to output this information in for example Slack so we know if the alert is new or repeated, eg inside the timeframe of repeat_interval setting.

I was hoping this would be available in the .Status data however it seems like that can only contain "firing" or "resolved".

Would this be possible to implement somehow or is it already available?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name a source file or test. Start by tracing how Alertmanager builds notification template data, focusing on .Status, firing alerts, and repeat_interval; done means a template can distinguish new from repeated firing alerts and render that information in Slack.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.