prometheus / prometheus/alertmanager

Alertmanager sending duplicate notifications after 'resolved' notification when running with multiple replicas

Open
#4,008 17 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/support
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
I have VM (Victoriametrics) Alert running with Alertmanager, with 2 replicas each. I am ingesting the metric disk_usage with value > threshold value. As soon as I receive an email from alertmanager (which is after 20 minutes in my case, see below files), I am stopping the data ingestion, which stops the alert breach as well.

  • VM_rules.yaml
groups:
- name: test_group
  interval: 5m
  concurrency: 1
  rules:
    - alert: High Disk Usage
      expr: ((avg(disk_usage[5m]) by (instance)) > 95)
      for: 15m
      labels:
        severity: critical
  • alertmanager_config.yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: smtp.gmail.com:587
  smtp_auth_username: XXX
  smtp_auth_password: XXX
  smtp_from: XXX@ABC.com
route:
  group_by: [alert_id, instance]
  receiver: test-default
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: test-default
  email_configs:
  - to: XYZ@ABC.com
    send_resolved: true 

What did you expect to see?
Alertmanager should send a 'fired' notification and a 'resolved' notification after I stop data ingestion because breach is stopped.

What did you see instead? Under which circumstances?
Alertmanager sends a 'fired' notification, but when the ingestion/breach is stopped, it sends a 'resolved' notification along with another 'fired' notification. And, 'resolved' email for this duplicate 'fired' email comes either instantaneously or in the next group_interval. Also, sometimes I see arbitrarily multiple pairs of duplicate 'fired-resolved' mails even after breach is stopped.
This unexpected behaviour is seen only when alertmanager is running with multiple (>=2) replicas.

Environment

  • Alertmanager version:
    v.0.27.0

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with VM_rules.yaml and alertmanager_config.yaml using two Alertmanager replicas, then compare the fired and resolved notifications after ingestion stops. Done means the configured setup sends one fired notification and one resolved notification, without duplicate pairs or extra notifications.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.