aptible / aptible/supercronic

Initialize prometheus counters with 0

Open
#83 1 comment 6 reactions 0 assignees View on GitHub
question
Dominant language
Go
Stars
2.6k
Forks
147
PR merge metrics
No merged PRs in 30d

Description

Hello,

i'm trying to create an alert based on supercronic prometheus metrics that will notify me if a job execution fails. If i have exactly one failed execution of a cron job the alert doesn't fire. Reason seems to be that `supercronic_failed_executions` counter isn't initialized when the process starts so the metric isn't available in prometheus until at least one failure. And when the first failure happens i cannot detect a change because there's nothing to compare to, the metric has a single value. With at least 2 failures the alert fires but i would really like to know about the first failure too. Initializing all counters with 0 will solve this. Also see the first part of this blog post https://blog.doit-intl.com/making-peace-with-prometheus-rate-43a3ea75c4cf.

My alert rule is `supercronic_failed_executions - supercronic_failed_executions offset 10m > 1`. Doing something like `supercronic_failed_executions > 1` will work but isn't useful because such alert will fire until the process is restarted even if there are successful executions afterwards.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by locating where the supercronic_failed_executions Prometheus counter is created and exposed, then inspect how counters are registered at process startup. Done means the counter is present with value 0 before any failure and the first failure can be detected by the stated alert rule.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.