prometheus / prometheus/alertmanager

Repeated notifications sooner than `repeat_interval`

Open
#2,320 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
Created an alert and a route to Slack with the following config:

group_wait: 10s
group_interval: 5m
repeat_interval: 24h

What did you expect to see?
One alert every 24h for the same alert

What did you see instead? Under which circumstances?
A repeated alert being sent after some amount of group_interval has passed (5m, 10m, 15m, etc). What I believe is happening is that a new data point appears after some amount of time < group_interval, but it then has resolved itself after the full duration of the group_interval. Additionally I have confirmed that there are no changes in the labels for the alert; I have viewed the query that triggers the alert and the labels stay consistent the whole time.

For example, we have an alert that is triggered when a kubernetes pod is in the state: Error:

  • at 10:50.00 an alert is triggered with the labels pod: foo, state: Error
  • at 10:50.10 the alert is sent to Slack after waiting the group_wait
  • at 10:52.00 a new data point appears with the labels pod: bar, state: Error. An alert is not sent yet because the group_interval of 5m has not passed yet.
  • at 10:52.30 the pod: bar is no longer in state: Error (note, pod: foo is still in state: Error
  • at 10:55.10 the alert is sent out but it only contains the info for pod: foo, state: Error because the pod: bar is no longer in state: Error

I would expect to see either:

  • No new alert being sent out at 10:55.10
  • A new alert being sent out at 10:55.10 that contains info about pod: bar

Environment

  • System information:
$ uname -srm
Linux 5.4.38-17.76.amzn2.x86_64 x86_64
  • Alertmanager version:
$ alertmanager --version
alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user:       root@dee35927357f
  build date:       20200617-08:54:02
  go version:       go1.14.4
  • Prometheus version:
$ prometheus --version
prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
  build user:       root@dd72efe1549d
  build date:       20200626-09:02:20
  go version:       go1.14.4
  • Alertmanager configuration file:
global: {}
receivers:
- name: slack-receiver
  slack_configs:
  - api_url: <url>
    channel: <channel>
    send_resolved: true
route:
  group_by:
  - alertname
  group_interval: 5m
  group_wait: 10s
  receiver: slack-receiver
  repeat_interval: 24h
  routes: []

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Alertmanager's notification scheduling around group_interval and repeat_interval, using the reported pod label lifecycle as the reproduction case. Trace why a resolved alert group member causes a notification at 5-minute intervals; done means unchanged labels do not trigger an early repeat and the notification behavior matches one of the stated expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.