prometheus / prometheus/alertmanager
Repeated notifications sooner than `repeat_interval`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
Created an alert and a route to Slack with the following config:
group_wait: 10s
group_interval: 5m
repeat_interval: 24h
What did you expect to see?
One alert every 24h for the same alert
What did you see instead? Under which circumstances?
A repeated alert being sent after some amount of group_interval has passed (5m, 10m, 15m, etc). What I believe is happening is that a new data point appears after some amount of time < group_interval, but it then has resolved itself after the full duration of the group_interval. Additionally I have confirmed that there are no changes in the labels for the alert; I have viewed the query that triggers the alert and the labels stay consistent the whole time.
For example, we have an alert that is triggered when a kubernetes pod is in the state: Error:
- at 10:50.00 an alert is triggered with the labels
pod: foo, state: Error - at 10:50.10 the alert is sent to Slack after waiting the
group_wait - at 10:52.00 a new data point appears with the labels
pod: bar, state: Error. An alert is not sent yet because thegroup_intervalof 5m has not passed yet. - at 10:52.30 the
pod: baris no longer instate: Error(note,pod: foois still instate: Error - at 10:55.10 the alert is sent out but it only contains the info for
pod: foo, state: Errorbecause thepod: baris no longer instate: Error
I would expect to see either:
- No new alert being sent out at 10:55.10
- A new alert being sent out at 10:55.10 that contains info about
pod: bar
Environment
- System information:
$ uname -srm
Linux 5.4.38-17.76.amzn2.x86_64 x86_64
- Alertmanager version:
$ alertmanager --version
alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
build user: root@dee35927357f
build date: 20200617-08:54:02
go version: go1.14.4
- Prometheus version:
$ prometheus --version
prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626-09:02:20
go version: go1.14.4
- Alertmanager configuration file:
global: {}
receivers:
- name: slack-receiver
slack_configs:
- api_url: <url>
channel: <channel>
send_resolved: true
route:
group_by:
- alertname
group_interval: 5m
group_wait: 10s
receiver: slack-receiver
repeat_interval: 24h
routes: []
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Alertmanager's notification scheduling around group_interval and repeat_interval, using the reported pod label lifecycle as the reproduction case. Trace why a resolved alert group member causes a notification at 5-minute intervals; done means unchanged labels do not trigger an early repeat and the notification behavior matches one of the stated expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, prometheus
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100