prometheus / prometheus/alertmanager

Group Timings Visibility

Open
#3,294 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

When designing an alerting pipeline, it's important for an operator to understand how the semantics of the system map their understanding.

The Alertmanager has some knobs such as:

  • Group Interval (Global and routing specific)
  • Repeat Interval (Global and routing specific)
  • Group Wait (Global and routing specific)
  • HA_peer timeout (HA mode only + per replica)

Currently, there are no metrics or visibility into any of these - so operators are left on "waiting" to see what happens or checking to see if any of the log lines make sense.

I think we can do better and provide visibility into the current state of all groups, so that operators can answer questions such as:

  • What this is alert group was last evaluated?
  • What state are these alert groups that I see?
  • Which replica is firing what?
  • What is the state of this replica?

I recognise that these answers are not simple when we think about alertmanager in HA mode and there's a chance of divergence but I think we can do better than what we currently do at the moment.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating how group timing and HA state are managed, then define the visibility needed for group evaluation, replica state, and waiting intervals; done requires an agreed scope and tests for the proposed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.