prometheus / prometheus/alertmanager
"target-only" inhibition rules only work if there're other alerts firing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
Assume I have a setup where I have "trigger alerts" that are basically only used to inhibit other alerts or I want to inhibit whole categories of (noisy) alerts. I have found "target-only" inhibition rules (i.e. rules with only target_matchers but no source_matchers or equal) to be a great "loop hole" for simple "self-inhibiting" rules:
inhibit_rules:
[...]
- target_matchers: [ severity="notify" ] # i.e. trigger alerts
- target_matchers:
- device_role="laptop"
- severity="warning"
- alertname=~"HostSwapIsFillingUp|HostOutOfMemory|PrometheusTargetMissing":
[...]
What did you expect to see?
These should inhibit the matched target alerts (since there's no overlap with source_matchers the exception for self-inhibiting rules does not apply).
What did you see instead? Under which circumstances?
These (target-only) inhibition rules worked as expected (not sure if intended or not 😉 ) for 6-8 weeks, but started experiencing "racy" behavior. I couldn't exactly pinpoint when it started, but my observation over the last weeks was:
if there're only alerts firing which should be inhibited by these target-only rules they will not be inhibited. Once a non-inhibited alert fires the target-only inhibition rules are applied again. This is most obvious when there're only one or two alerts in flight.
All other inhibition rules seem to work without problems.
Environment
- System information:
Docker version 20.10.11, build dea9396 on Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-41-generic x86_64)
- Alertmanager version:
$ docker exec alertmanager alertmanager --version
alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b17771a57cfd4c4a51be370ab930a4d7d54)
build user: root@e21a959be8d2
build date: 20210825-10:48:55
go version: go1.16.7
platform: linux/amd64
- Prometheus version:
$ docker exec prometheus prometheus --version
prometheus, version 2.31.1 (branch: HEAD, revision: 411021ada9ab41095923b8d2df9365b632fd40c3)
build user: root@9419c9c2d4e0
build date: 20211105-20:35:02
go version: go1.17.3
platform: linux/amd64
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported configuration with only alerts matching target-only inhibition rules, then repeat with one non-inhibited alert present. Trace Alertmanager's inhibition evaluation and add a regression test covering both cases. Done means target alerts are inhibited consistently even when no non-inhibited alert is firing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100