prometheus / prometheus/alertmanager
Alertmanager sending duplicate notifications after 'resolved' notification when running with multiple replicas
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
I have VM (Victoriametrics) Alert running with Alertmanager, with 2 replicas each. I am ingesting the metric disk_usage with value > threshold value. As soon as I receive an email from alertmanager (which is after 20 minutes in my case, see below files), I am stopping the data ingestion, which stops the alert breach as well.
- VM_rules.yaml
groups:
- name: test_group
interval: 5m
concurrency: 1
rules:
- alert: High Disk Usage
expr: ((avg(disk_usage[5m]) by (instance)) > 95)
for: 15m
labels:
severity: critical
- alertmanager_config.yaml
global:
resolve_timeout: 5m
smtp_smarthost: smtp.gmail.com:587
smtp_auth_username: XXX
smtp_auth_password: XXX
smtp_from: XXX@ABC.com
route:
group_by: [alert_id, instance]
receiver: test-default
group_interval: 5m
repeat_interval: 12h
receivers:
- name: test-default
email_configs:
- to: XYZ@ABC.com
send_resolved: true
What did you expect to see?
Alertmanager should send a 'fired' notification and a 'resolved' notification after I stop data ingestion because breach is stopped.
What did you see instead? Under which circumstances?
Alertmanager sends a 'fired' notification, but when the ingestion/breach is stopped, it sends a 'resolved' notification along with another 'fired' notification. And, 'resolved' email for this duplicate 'fired' email comes either instantaneously or in the next group_interval. Also, sometimes I see arbitrarily multiple pairs of duplicate 'fired-resolved' mails even after breach is stopped.
This unexpected behaviour is seen only when alertmanager is running with multiple (>=2) replicas.
Environment
- Alertmanager version:
v.0.27.0
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the behavior with VM_rules.yaml and alertmanager_config.yaml using two Alertmanager replicas, then compare the fired and resolved notifications after ingestion stops. Done means the configured setup sends one fired notification and one resolved notification, without duplicate pairs or extra notifications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100