prometheus / prometheus/alertmanager
PagerDuty alerts are stuck if I mute the alert after it's firing but before it's resolved
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
- Have an Alertmanager setup that has PagerDuty as a receiver
- Do something that triggers an alert that is routed to PagerDuty receiver
- Mute an alert
- Fix it so it'd get resolved
- Alertmanager doesn't send a notification on a resolved alert as it's muted
- PagerDuty doesn't know it's fixed so it's stuck there unless I manually go to PagerDuty app and resolve it there
- Mute expires, so I end up with a case when the firing is no more firing, but PagerDuty thinks it's still firing and bugs me until I resolve it via PagerDuty app
The procedure I describe above is something I do constantly (as in, something breaks, I mute it so it won't annoy me with notifications as I know it's broken and I am fixing it, it's fixed before the mute is expired, and the alert is stuck in PagerDuty). For Telegram it seems ok, as nothing wrong will happen if I won't receive a resolve notification, but for PagerDuty (and I think for other similar services like OpsGenie, but I cannot verify that as I do not use it) it might be annoying as they might have their own set of alerting rules, like pinging a person every hour or sending them a push notification/SMS/call once the alert is firing.
What did you expect to see?
Alert is resolved in PagerDuty.
What did you see instead? Under which circumstances?
Alert is not resolved and is stuck, although is fixed.
Environment
-
System information:
Linux 5.15.0-91-generic x86_64 -
Alertmanager version:
alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
build user: root@df8d7debeef4
build date: 20230824-11:11:58
go version: go1.20.7
platform: linux/amd64
tags: netgo
- Prometheus version:
Pretty sure it's not relevant, but:
prometheus, version 2.48.1 (branch: HEAD, revision: 63894216648f0d6be310c9d16fb48293c45c9310)
build user: root@71f108ff5632
build date: 20231208-23:33:22
go version: go1.21.5
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
- Alertmanager configuration file:
Something similar to this (I removed some parts that are not relevant, like inhibit rules):
global:
pagerduty_url: 'https://events.eu.pagerduty.com'
route:
receiver: 'telegram'
group_wait: 10s
group_by: ['host', 'hosting', 'datacenter', 'alertname']
repeat_interval: 1h
routes:
- receiver: 'telegram'
match_re:
severity: critical|warning
continue: true
- receiver: 'pagerduty'
group_by: ['...']
match_re:
severity: critical
continue: true
receivers:
- name: 'telegram'
telegram_configs:
- send_resolved: true
message: '{{ template "telegram.text" . }}'
bot_token: 'xxxx'
chat_id: 123
- name: 'pagerduty'
pagerduty_configs:
- service_key: xxxx
templates:
- /etc/alertmanager/templates/template.tmpl
- Prometheus configuration file:
insert configuration here (if relevant to the issue)
- Logs:
insert Prometheus and Alertmanager logs relevant to the issue here
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied mute-then-resolve sequence with Alertmanager's PagerDuty receiver and trace how notifications are handled while the alert is muted. Compare the behavior with the expected resolved state and use the existing notification tests, if located, to capture the case where PagerDuty must learn that the alert has resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100