prometheus / prometheus/alertmanager
Feature request: Add group creation time to group_by hash
Open
Nobody has claimed this yet.
kind/enhancement
stale
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
- Alert fired, started flapping
- Ops person manually silenced the alert
- Ops person also acknowledged the alert in Opsgenie
What did you expect to see?
- A new batch of alerts should not be grouped into an already resolved group
What did you see instead? Under which circumstances?
- Alert resolved, alertmanager did not send a notification to Opsgenie due to #226
- A week later, new hosts started alerting, but were grouped into the same already-acknowledged incident in Opsgenie
- Full datacenter outage occurs due to missed alerts
Environment
- Alertmanager version:
alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)
build user: root@265f14f5c6fc
build date: 20220325-09:31:33
go version: go1.17.8
platform: linux/amd64
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the group_by hash is generated and how grouped alerts are sent to Opsgenie. Confirm how group creation time could distinguish a new batch from an already resolved group, then add regression coverage for the acknowledged-incident scenario. Done means later alerts are not grouped into the old resolved group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100