prometheus / prometheus/alertmanager

group_by: [alertname, alertstate] doesn't work as it should

Open
#2,334 27 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/notify kind/enhancement kind/more-info-needed
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
I used

group_by:
  - alertname
  - alertstate

config and got multiple alerts (both firing and resolved) mixed together in a single notification.

What did you expect to see?
Separate notifications: one for firing alerts and the other for resolved ones.

What did you see instead? Under which circumstances?
A single notification with multiple alerts (both firing and resolved) mixed together.

Environment
Yes.

  • System information:

Linux 3.10.0-1127.el7.x86_64 x86_64

  • Alertmanager version:

alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
build user: root@dee35927357f
build date: 20200617-08:54:02
go version: go1.14.4

  • Prometheus version:

prometheus, version 2.19.2 (branch: HEAD, revision: c448ada63d83002e9c1d2c9f84e09f55a61f0ff7)
build user: root@dd72efe1549d
build date: 20200626-09:02:20
go version: go1.14.4

  • Alertmanager configuration file:
global:
  resolve_timeout: 5m
  smtp_from: x@y.com
  smtp_smarthost: 10.0.0.2:25
  smtp_require_tls: false

route:
  group_by:
    - alertname
    - alertstate
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 3h
  receiver: topguys

receivers:
  - name: topguys
    email_configs:
      - to: z@y.com
        send_resolved: true

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with the supplied Alertmanager route configuration, especially group_by containing alertname and alertstate, with send_resolved enabled. Trace how the route groups firing and resolved alerts, then add coverage for the reported configuration; done means firing and resolved alerts arrive in separate notifications.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.