goauthentik / goauthentik/helm

Bundled Prometheus recording rules reference non-existent metric names (doubled `_total` suffix) — all rules produce 0 samples

Open Beginner friendly
#491 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mustache
Stars
183
Forks
69
Avg merge
14h 21m
Merged PRs (30d)
8

Description

## Summary
The recording rules shipped with the chart (`prometheus.rules`) query django-prometheus metrics by their Python source names, e.g. `django_http_responses_total_by_status`.
However, the Prometheus Python client appends the conventional `_total` suffix to counter samples at exposition time, so the series actually scraped is
`django_http_responses_total_by_status_total` (double suffix, since django-prometheus already embeds `_total` mid-name).

As a result, every recording rule in the bundled groups ("Aggregate request counters", "Aggregate model operations", "Aggregate database operations") matches no series and has produced 0 samples. This goes unnoticed unless something meta-monitors recording-rule output (in my case, VictoriaMetrics' vmalert alerted on rules producing no samples for 30m).

## Environment
- authentik: 2026.5.3
- Chart: 2026.5.3
- Metrics pipeline: victoria-metrics-k8s-stack (vmagent/vmalert, PrometheusRule
converted via VM operator)
**not VM-specific**: the name mismatch exists at the exposition endpoint itself, before any scraper is involved (see repro).

## Reproduction

Rule as rendered by the chart:
```console
$ kubectl -n authentik get vmrule -A -o yaml | grep -A2 'by_status:sum_rate30s'
record: job:django_http_responses_total_by_status:sum_rate30s
- expr: sum(rate(django_http_responses_total_by_status_name_method[30s])) by
(job,status,name,method)
```

What the app actually exposes:
```console
$ kubectl -n authentik exec deploy/authentik-server -- curl -s localhost:9300/metrics | grep -m3 by_status
# HELP django_http_responses_total_by_status_total Count of responses by status.
# TYPE django_http_responses_total_by_status_total counter
django_http_responses_total_by_status_total{status="200"} 394.0
```

The rule queries `..._by_status` / `..._by_status_name_method`; the exposed (and therefore stored) series carry a trailing `_total`.
`rate()` over a non-existent series returns empty, so the recording rules emit nothing.
The same mismatch applies to the other counter-based rules, e.g. `django_http_responses_body_total_bytes`, `django_http_responses_total_by_charset`, `django_http_responses_total_by_templatename`, `django_model_inserts/updates/deletes_total`, `django_db_new_connection_errors_total`, `django_db_execute_errors_total` - all are exposed with an additional `_total` suffix.

## Root cause
prometheus/client_python appends `_total` to counter names on exposition (since 0.4.0).
django-prometheus defines counters whose declared names already contain `_total` (e.g. `django_http_responses_total_by_status`), so the scraped series is `django_http_responses_total_by_status_total`. The chart's rule expressions were written against the declared names, not the exposed sample names, so they have never matched the scraped data.

[Counter exposition behavior (official client_python docs)](https://prometheus.github.io/client_python/instrumenting/counter/)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the chart's prometheus.rules template and compare its expressions with the metrics shown by the /metrics reproduction. Update the counter references across the "Aggregate request counters," "Aggregate model operations," and "Aggregate database operations" groups, then verify the rendered rules use the exposed trailing _total series names and produce samples.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, prometheus
Domain
devops, observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.