prometheus / prometheus/alertmanager

"target-only" inhibition rules only work if there're other alerts firing

Open
#2,783 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?

Assume I have a setup where I have "trigger alerts" that are basically only used to inhibit other alerts or I want to inhibit whole categories of (noisy) alerts. I have found "target-only" inhibition rules (i.e. rules with only target_matchers but no source_matchers or equal) to be a great "loop hole" for simple "self-inhibiting" rules:

inhibit_rules:
[...]
- target_matchers: [ severity="notify" ]  # i.e. trigger alerts
- target_matchers:
    - device_role="laptop"
    - severity="warning"
    - alertname=~"HostSwapIsFillingUp|HostOutOfMemory|PrometheusTargetMissing":
[...]

What did you expect to see?

These should inhibit the matched target alerts (since there's no overlap with source_matchers the exception for self-inhibiting rules does not apply).

What did you see instead? Under which circumstances?

These (target-only) inhibition rules worked as expected (not sure if intended or not 😉 ) for 6-8 weeks, but started experiencing "racy" behavior. I couldn't exactly pinpoint when it started, but my observation over the last weeks was:
if there're only alerts firing which should be inhibited by these target-only rules they will not be inhibited. Once a non-inhibited alert fires the target-only inhibition rules are applied again. This is most obvious when there're only one or two alerts in flight.

All other inhibition rules seem to work without problems.

Environment

  • System information:

Docker version 20.10.11, build dea9396 on Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-41-generic x86_64)

  • Alertmanager version:
$ docker exec alertmanager alertmanager --version
alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b17771a57cfd4c4a51be370ab930a4d7d54)
  build user:       root@e21a959be8d2
  build date:       20210825-10:48:55
  go version:       go1.16.7
  platform:         linux/amd64
  • Prometheus version:
$ docker exec prometheus prometheus --version
prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)
  build user:       root@9419c9c2d4e0
  build date:       20211105-20:35:02
  go version:       go1.17.3
  platform:         linux/amd64

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported configuration with only alerts matching target-only inhibition rules, then repeat with one non-inhibited alert present. Trace Alertmanager's inhibition evaluation and add a regression test covering both cases. Done means target alerts are inhibited consistently even when no non-inhibited alert is firing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.