goauthentik / goauthentik/helm
Bundled Prometheus recording rules reference non-existent metric names (doubled `_total` suffix) — all rules produce 0 samples
- Dominant language
- Mustache
- Stars
- 183
- Forks
- 69
- Avg merge
- 14h 21m
- Merged PRs (30d)
- 8
Description
## Summary
The recording rules shipped with the chart (`prometheus.rules`) query django-prometheus metrics by their Python source names, e.g. `django_http_responses_total_by_status`.
However, the Prometheus Python client appends the conventional `_total` suffix to counter samples at exposition time, so the series actually scraped is
`django_http_responses_total_by_status_total` (double suffix, since django-prometheus already embeds `_total` mid-name).
As a result, every recording rule in the bundled groups ("Aggregate request counters", "Aggregate model operations", "Aggregate database operations") matches no series and has produced 0 samples. This goes unnoticed unless something meta-monitors recording-rule output (in my case, VictoriaMetrics' vmalert alerted on rules producing no samples for 30m).
## Environment
- authentik: 2026.5.3
- Chart: 2026.5.3
- Metrics pipeline: victoria-metrics-k8s-stack (vmagent/vmalert, PrometheusRule
converted via VM operator)
**not VM-specific**: the name mismatch exists at the exposition endpoint itself, before any scraper is involved (see repro).
## Reproduction
Rule as rendered by the chart:
```console
$ kubectl -n authentik get vmrule -A -o yaml | grep -A2 'by_status:sum_rate30s'
record: job:django_http_responses_total_by_status:sum_rate30s
- expr: sum(rate(django_http_responses_total_by_status_name_method[30s])) by
(job,status,name,method)
```
What the app actually exposes:
```console
$ kubectl -n authentik exec deploy/authentik-server -- curl -s localhost:9300/metrics | grep -m3 by_status
# HELP django_http_responses_total_by_status_total Count of responses by status.
# TYPE django_http_responses_total_by_status_total counter
django_http_responses_total_by_status_total{status="200"} 394.0
```
The rule queries `..._by_status` / `..._by_status_name_method`; the exposed (and therefore stored) series carry a trailing `_total`.
`rate()` over a non-existent series returns empty, so the recording rules emit nothing.
The same mismatch applies to the other counter-based rules, e.g. `django_http_responses_body_total_bytes`, `django_http_responses_total_by_charset`, `django_http_responses_total_by_templatename`, `django_model_inserts/updates/deletes_total`, `django_db_new_connection_errors_total`, `django_db_execute_errors_total` - all are exposed with an additional `_total` suffix.
## Root cause
prometheus/client_python appends `_total` to counter names on exposition (since 0.4.0).
django-prometheus defines counters whose declared names already contain `_total` (e.g. `django_http_responses_total_by_status`), so the scraped series is `django_http_responses_total_by_status_total`. The chart's rule expressions were written against the declared names, not the exposed sample names, so they have never matched the scraped data.
[Counter exposition behavior (official client_python docs)](https://prometheus.github.io/client_python/instrumenting/counter/)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the chart's prometheus.rules template and compare its expressions with the metrics shown by the /metrics reproduction. Update the counter references across the "Aggregate request counters," "Aggregate model operations," and "Aggregate database operations" groups, then verify the rendered rules use the exposed trailing _total series names and produce samples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, prometheus
- Domain
- devops, observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100