prometheus / prometheus/alertmanager

PagerDuty alerts are stuck if I mute the alert after it's firing but before it's resolved

Open
#3,664 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?

  1. Have an Alertmanager setup that has PagerDuty as a receiver
  2. Do something that triggers an alert that is routed to PagerDuty receiver
  3. Mute an alert
  4. Fix it so it'd get resolved
  5. Alertmanager doesn't send a notification on a resolved alert as it's muted
  6. PagerDuty doesn't know it's fixed so it's stuck there unless I manually go to PagerDuty app and resolve it there
  7. Mute expires, so I end up with a case when the firing is no more firing, but PagerDuty thinks it's still firing and bugs me until I resolve it via PagerDuty app

The procedure I describe above is something I do constantly (as in, something breaks, I mute it so it won't annoy me with notifications as I know it's broken and I am fixing it, it's fixed before the mute is expired, and the alert is stuck in PagerDuty). For Telegram it seems ok, as nothing wrong will happen if I won't receive a resolve notification, but for PagerDuty (and I think for other similar services like OpsGenie, but I cannot verify that as I do not use it) it might be annoying as they might have their own set of alerting rules, like pinging a person every hour or sending them a push notification/SMS/call once the alert is firing.

What did you expect to see?
Alert is resolved in PagerDuty.

What did you see instead? Under which circumstances?
Alert is not resolved and is stuck, although is fixed.

Environment

  • System information:
    Linux 5.15.0-91-generic x86_64

  • Alertmanager version:

alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@df8d7debeef4
  build date:       20230824-11:11:58
  go version:       go1.20.7
  platform:         linux/amd64
  tags:             netgo
  • Prometheus version:

Pretty sure it's not relevant, but:

prometheus, version 2.48.1 (branch: HEAD, revision: 63894216648f0d6be310c9d16fb48293c45c9310)
  build user:       root@71f108ff5632
  build date:       20231208-23:33:22
  go version:       go1.21.5
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
  • Alertmanager configuration file:
    Something similar to this (I removed some parts that are not relevant, like inhibit rules):
global:
  pagerduty_url: 'https://events.eu.pagerduty.com'

route:
  receiver: 'telegram'
  group_wait: 10s
  group_by: ['host', 'hosting', 'datacenter', 'alertname']
  repeat_interval: 1h

  routes:
    - receiver: 'telegram'
      match_re:
        severity: critical|warning
      continue: true

    - receiver: 'pagerduty'
      group_by: ['...']
      match_re:
        severity: critical
      continue: true

receivers:
  - name: 'telegram'
    telegram_configs:
      - send_resolved: true
        message: '{{ template "telegram.text" . }}'
        bot_token: 'xxxx'
        chat_id: 123

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: xxxx

templates:
  - /etc/alertmanager/templates/template.tmpl

  • Prometheus configuration file:
insert configuration here (if relevant to the issue)
  • Logs:
insert Prometheus and Alertmanager logs relevant to the issue here

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied mute-then-resolve sequence with Alertmanager's PagerDuty receiver and trace how notifications are handled while the alert is muted. Compare the behavior with the expected resolved state and use the existing notification tests, if located, to capture the case where PagerDuty must learn that the alert has resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.