Initialize prometheus counters with 0
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 147
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
i'm trying to create an alert based on supercronic prometheus metrics that will notify me if a job execution fails. If i have exactly one failed execution of a cron job the alert doesn't fire. Reason seems to be that `supercronic_failed_executions` counter isn't initialized when the process starts so the metric isn't available in prometheus until at least one failure. And when the first failure happens i cannot detect a change because there's nothing to compare to, the metric has a single value. With at least 2 failures the alert fires but i would really like to know about the first failure too. Initializing all counters with 0 will solve this. Also see the first part of this blog post https://blog.doit-intl.com/making-peace-with-prometheus-rate-43a3ea75c4cf.
My alert rule is `supercronic_failed_executions - supercronic_failed_executions offset 10m > 1`. Doing something like `supercronic_failed_executions > 1` will work but isn't useful because such alert will fire until the process is restarted even if there are successful executions afterwards.
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named. Start by locating where the supercronic_failed_executions Prometheus counter is created and exposed, then inspect how counters are registered at process startup. Done means the counter is present with value 0 before any failure and the first failure can be detected by the stated alert rule.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100