prometheus / prometheus/alertmanager

Alertmanager loses in-memory alerts when it restarts

Open
#2,370 4 comments 13 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/friction stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
Alertmanager stores fingerprints of current alerts in the memory send via clients. It stores nf-logs and silences in the disk(if configured).
When alertmanger starts it reads nf-logs and silences from the the disk. The in-memory store is recreated when the alertmanager restarts. Now, if the client has stopped sending alerts to alertmanager while alertmanager was down, the AM will not be able to send resolve notifications to receivers.

What did you expect to see?
AM should be able to send resolve notification after it restarts.

Can we also store the in-memory alerts onto the disk via snapshot(similar to nf-logs and silences) and/or while graceful shutdown and recover them from disk when AM restarts?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing Alertmanager's in-memory alert store and the existing disk persistence paths for notification logs and silences. Investigate how startup and graceful shutdown handle state, then define and test recovery so alerts that stop arriving while Alertmanager is down can still produce resolve notifications after restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.