prometheus / prometheus/alertmanager
Receiving alert more often then repeat interval
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
We have been working on cleaning up our alerts recently. And we are finding that for some alerts, the same alerts are firing over and over. And I mean like a repeat. The StartsAt times and all other fields are an exact match.
We have multiple clusters and each is setup with it's own alertmanager. We use a template to configure them so they are all setup the same (and I checked the config on the pods). But the alerts that repeat seem to be one-offs. Like the watchdog alert. Only one cluster has it repeating over and over again more often than the repeat_interval. Then we have a Service500Error error alert watching linkerd status code 500's. In a given cluster, only one of the service 500's repeats to often, the others don't. And the cluster where that happens is different than the cluster that repeats the watchdog.
So it seems like individual alerts are somehow behaving differently than any others. That leads me to believe it isn't a general configuration issue. Both of these alerts are kind of "always" firing type alerts. That is why the Service500Error is set to repeat every 120h to match the max retention. The history of that alert is a whole long story, but shouldn't really matter.
Environment
We use kube-prometheus-stack version 44.3.0
which has version 0.25.0 of alertmanager.
We are running on kubernetes 1.21 in aws and GCP in managed clusters.
* Alertmanager configuration file:
global:
resolve_timeout: 5m
route:
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: "null"
group_by:
- job
routes:
- receiver: "null"
matchers:
- alertname=DeadMansSwitch
- receiver: slack
matchers:
- alertname=Service500Error
repeat_interval: 120h
- receiver: slack
- Logs:
There is this message in the logs.
http: URL query contains semicolon, which is no longer a supported separator; parts of the query may be stripped when parsed; see golang.org/issue/25192
Other than that nothing much in the logs. Just confirmation that the config was read. I don't really know what that error in the log is about. But can't really see how it would be related.
Overall, I can't see how this is enough information to debug the issue on your end. But I don't know how to get more information to help debug the issue. I am happy to do more if I can get some ideas on how to get more info.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied Alertmanager route configuration, the kube-prometheus-stack and Alertmanager versions, and the reported log message. Reproduce or narrow down why individual alerts repeat before the configured interval; done means identifying the cause or the additional evidence needed to diagnose it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100