Emit metric for failure to step jobs
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 70
- Forks
- 16
- Avg merge
- 4h 36m
- Merged PRs (30d)
- 30
Description
It's possible for Janus to fail to step a collection job because it notices that there are pending aggregation jobs, or that there are relevant reports that have not yet been scheduled into a job. When this happens, we increment step_attempts and try again later. But then we have no indication of why we released the job, except that if it was due to failure to make a request to the helper or a DB problem, we'll log an error.
We should emit a metric from Janus that increments when we release a job and increment the step attempts counter. We could then label with a reason enum (pending_agg_jobs | unaggregated_report | http_request_falure| etc.). It can't be labeled by task or job ID to avoid high cardinality metrics, but at least then we'd notice a correlation between long collection job completion times and an uptick in that graph.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Janus code path that releases a collection job and increments step_attempts. Trace the existing reasons for release, including pending aggregation jobs, unaggregated reports, helper HTTP failures, and database problems. Done means a metric is emitted for these failures with bounded reason labels and without task or job IDs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100