prometheus / prometheus/alertmanager

Allow mutating alerts on pipeline stages

Open
#4,361 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

Currently the alertmanager notify stages allow things like dedupe, group by amongst other things. What we have observed is that mutating of alerts carry a lot of interesting possibilities such as:

  • attach exemplars to each alert via annotations and paint them on the corresponding receiver
  • group alerts by a certain grouping key and attach a root cause which can be fetched from a certain external API

currently alerts are dispatched parallel to each receiver which means that any stage is concurrently invoked causing map updates to annotations to cause a panic.

some of the things that would be useful:

  • support mutation stages that are common across all receivers (we would hate to see different sample traces across various receivers)
  • support grouping of alerts post mutation to facilitate more complex things

Currently things like AlertStoreCallback can be used to achieve some of this but said callback blocks the POST/PUT API which can cause a slow down on the rule manager.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the alert notification pipeline and the AlertStoreCallback path, then trace how alerts are dispatched in parallel to receivers. Define how shared mutation stages, post-mutation grouping, and safe annotation updates should work; done means the design supports these behaviors without blocking the POST/PUT API or causing concurrent map-update panics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.