prometheus / prometheus/alertmanager
Alertmanager loses in-memory alerts when it restarts
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
Alertmanager stores fingerprints of current alerts in the memory send via clients. It stores nf-logs and silences in the disk(if configured).
When alertmanger starts it reads nf-logs and silences from the the disk. The in-memory store is recreated when the alertmanager restarts. Now, if the client has stopped sending alerts to alertmanager while alertmanager was down, the AM will not be able to send resolve notifications to receivers.
What did you expect to see?
AM should be able to send resolve notification after it restarts.
Can we also store the in-memory alerts onto the disk via snapshot(similar to nf-logs and silences) and/or while graceful shutdown and recover them from disk when AM restarts?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing Alertmanager's in-memory alert store and the existing disk persistence paths for notification logs and silences. Investigate how startup and graceful shutdown handle state, then define and test recovery so alerts that stop arriving while Alertmanager is down can still produce resolve notifications after restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100