prometheus / prometheus/alertmanager
Add batch routing test file support with inhibition assertions to `amtool config routes test`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
Problem
amtool currently provides two tools for validating alertmanager configuration:
amtool check-configvalidates syntax and structure. It cannot tell you whether an alert reaches the right receiver.amtool config routes testtests a single alert against the routing tree interactively. It is useful for manual debugging but has two significant limitations:- It does not support batch test files — you cannot define a suite of routing expectations and run them all at once.
- It has no notion of inhibition. You cannot assert that a warning alert is suppressed when a critical is firing.
The failure modes these gaps allow are silent and high-impact:
- Wrong receiver:
SomeAlertwithteam=backendends up in the frontend Slack channel because a route was added above the team-based route withoutcontinue: false. The config is syntactically valid;amtool check-configpasses; the routing logic is just wrong. - Broken inhibition: A warning fires even though a critical is active for the same alert, flooding an incident channel with noise. Or inhibition is over-broad and silences warnings that should not be suppressed.
Both are semantic errors that cannot be caught today without either manual testing against a live alertmanager instance or a separate out-of-tree tool.
Proposed Feature
Extend amtool config routes test with a --test-file flag that accepts a YAML file containing a list of named test cases. Each test case fires one or more alerts together and asserts per-alert expectations: the expected receiver list, or that the alert is inhibited.
Running all alerts in a test case together is the key property that makes inhibition testing possible: a severity=critical alert can suppress a severity=warning alert within the same case, mirroring how a live alertmanager would behave.
Exit code 0 if all cases pass; exit code 1 if any fail, making it CI-friendly by default.
Proposed YAML Format
tests:
# Anything not matched by a specific route falls through to the default receiver.
- name: "Unmatched alert routes to default receiver"
alerts:
- labels:
alertname: SomeAlert
expected_receivers:
- default
# Watchdog is a synthetic heartbeat alert. It must not page anyone.
- name: "Watchdog alert routes to null receiver"
alerts:
- labels:
alertname: Watchdog
severity: critical
expected_receivers:
- "null"
# Team-based routing.
- name: "Team A alert routes to team-a-slack"
alerts:
- labels:
alertname: TeamAAlert
team: team-a
expected_receivers:
- team-a-slack
# Inhibition: a critical suppresses a warning with the same alertname.
# Both alerts are fired together so the inhibitor can evaluate the relationship.
- name: "critical suppresses warning with same alertname"
alerts:
- labels:
alertname: SomeAlert
severity: critical
expected_receivers:
- default
- labels:
alertname: SomeAlert
severity: warning
expected_inhibited: true
# Inhibition boundary: a critical for AlertOne does NOT suppress a warning
# for AlertTwo because the inhibit rule requires equal alertname.
- name: "critical does NOT suppress warning with different alertname"
alerts:
- labels:
alertname: AlertOne
severity: critical
expected_receivers:
- default
- labels:
alertname: AlertTwo
severity: warning
expected_receivers:
- default
Fields:
expected_receivers: ordered list of receiver names the alert must match. Order matters because alertmanager's routing order is significant whencontinue: trueis used.expected_inhibited: set totrueto assert the alert is suppressed. Omit (or leavefalse) otherwise. Do not set both on the same alert.
Expected CLI Output
PASS Unmatched alert routes to default receiver
PASS Watchdog alert routes to null receiver
PASS Team A alert routes to team-a-slack
PASS critical suppresses warning with same alertname
FAIL "critical does NOT suppress warning with different alertname"
alert {alertname=AlertTwo, severity=warning}:
expected: default
actual: (inhibited)
=== routing tests: 4 passed, 1 failed ===
Implementation Notes
Routing already works via the existing amtool code path. dispatch.NewRoute(cfg.Route, nil).Match(labelSet) returns the same receiver list a live alertmanager would produce.
Inhibition requires a minimal in-memory provider.Alerts implementation. The inhibitor is designed to work against a live alert store; the workaround is a fakeAlerts struct that serves a fixed set of alerts from a buffered channel:
func (f *fakeAlerts) Subscribe() provider.AlertIterator {
ch := make(chan *types.Alert, len(f.alerts))
for _, a := range f.alerts {
ch <- a
}
done := make(chan struct{})
return provider.NewAlertIterator(ch, done, nil)
}
The inhibitor is constructed with this fake provider, its Run() goroutine is started, and after a brief settle for it to process the alert feed, Mutes(labelSet) is called for each alert to check whether it is suppressed.
Test runner loop:
- For each test case, collect all alert label sets.
- Construct a
fakeAlertsprovider with all alerts in the case. - Start the inhibitor with
Run(). - For each alert in the case: check
inhibitor.Mutes(labels)first; if not inhibited, calldispatch.NewRoute(...).Match(labels). - Compare results against assertions; record PASS or FAIL.
- Tear down the inhibitor.
Inhibition is checked before receiver matching. Matching an inhibited alert to receivers is undefined behavior in a real alertmanager, so the test should assert inhibition explicitly via expected_inhibited: true.
For full context on this approach, see: https://dev.to/frosnerd/unit-testing-alertmanager-routing-and-inhibition-rules-1hj4
Relationship to Existing Issues
- #2776 — adds stdin support for
amtool config routes test(single alert, different scope) - #3003 — alert grouping test with amtool (related concept, grouping rather than routing/inhibition)
No existing issue covers batch test files together with inhibition assertions.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the existing amtool config routes test entry point and trace how dispatch.NewRoute(...).Match(labelSet) performs routing. Then inspect the provider.Alerts/provider.AlertIterator interfaces and inhibitor lifecycle described in the issue. Done means --test-file runs grouped cases, reports receiver and inhibition assertion results, and returns the specified CI-friendly exit codes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100