goauthentik / goauthentik/authentik

authentik_prometheus_tmp still accumulates PID files after Gunicorn worker recycling in 2026.8.1

Open
#25,876 1 comment 0 reactions 1 assignee Claimed by @BeryJu View on GitHub
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 1h
Merged PRs (30d)
644

Description

## Describe the bug

On authentik 2026.8.1, the server container continues to accumulate Prometheus
multiprocess files in `/dev/shm/authentik_prometheus_tmp` when Gunicorn workers
reach `max_requests` and recycle. Files belonging to exited worker PIDs remain,
so the default 64 MiB `/dev/shm` allocation grows continuously.

This appears related to #10768 and #16757. PR #20477 reduced files created by
short-lived policy evaluation processes, but the reproduction below shows that
Gunicorn worker recycling still leaves PID-specific counter, gauge, and
histogram files behind.

## How to reproduce

1. Run the Docker image `ghcr.io/goauthentik/server:2026.8.1` using Docker
Compose, with the default two web workers and the default 64 MiB `/dev/shm`.
No metrics configuration is required.
2. Record the initial allocation and file count:

```shell
df -k /dev/shm
du -sk /dev/shm/authentik_prometheus_tmp
find /dev/shm/authentik_prometheus_tmp -type f | wc -l
```

3. Send 2,500 sequential requests to an OIDC discovery endpoint. In this test,
requests were capped at 10 per second and all returned HTTP 200:

```shell
for i in $(seq 1 2500); do
curl -fsS -o /dev/null \
http://127.0.0.1:9000/application/o/example/.well-known/openid-configuration
sleep 0.1
done
```

4. Confirm in the server logs that both workers exited normally after reaching
their request limits and replacement workers started.
5. Repeat the allocation and file-count commands and list the directory.

Observed worker lifecycle:

```text
Worker exiting (pid: 110)
Booting worker with pid: 2192
Worker exiting (pid: 111)
Booting worker with pid: 2733
```

Observed results:

| State | `/dev/shm` allocated | Metrics directory | File count |
| --- | ---: | ---: | ---: |
| Before requests | 200 KiB | 196 KiB | 14 |
| After both workers recycled | 276 KiB | 272 KiB | 21 |

After recycling, the directory still contained files for exited PIDs 110 and
111 alongside files for replacement PIDs 2192 and 2733:

```text
counter_110.db
counter_111.db
counter_2192.db
counter_2733.db
gauge_all_110.db
gauge_all_111.db
gauge_all_2192.db
gauge_all_2733.db
histogram_110.db
histogram_111.db
histogram_2192.db
histogram_2733.db
```

The same 2026.8.1 image under normal production traffic grew from 14 files and
184 KiB allocated to 96 files and 1,612 KiB allocated in approximately 18
hours. That is about 80 KiB/hour, with no container restarts or abnormal worker
exits during the sample period.

## Expected behavior

Normal Gunicorn worker recycling should not cause unbounded growth in
`/dev/shm`. Metrics files should either use reusable stable worker identifiers
or be removed/compacted when a worker exits.

## Additional context

`lifecycle/gunicorn.conf.py` sets `MultiProcessValue(lambda:
worker._worker_id)` in `post_fork` and calls
`multiprocess.mark_process_dead(worker._worker_id)` in `worker_exit`. However,
the files observed during this test were named with the operating-system PIDs.
This may mean some collectors retain the PID-based `ValueClass` initialized
before `post_fork`, preventing the stable worker identifiers and exit cleanup
from covering them.

Before upgrading, the same deployment experienced an outage on 2025.12.4 with
7,079 repeated Gunicorn `SIGBUS` worker crashes and router errors such as
`failed to proxy to backend: EOF`. A container restart cleared `/dev/shm` and
restored service. The reboot erased the full pre-outage directory state, so the
historical outage cannot by itself prove that `/dev/shm` was exhausted; the
repeatable 2026.8.1 growth above is independent of that inference.

Increasing `shm_size` or periodically restarting the server container delays or
clears the accumulation but does not address its source.

Related:

- #10768
- #16757
- #20477
- #19369

## Deployment Method

Docker

## Version

2026.8.1

## Relevant log output

```shell
{"event":"Worker exiting (pid: 110)","level":"info","logger":"gunicorn.error"}
{"event":"Booting worker with pid: 2192","level":"info","logger":"gunicorn.error"}
{"event":"Worker exiting (pid: 111)","level":"info","logger":"gunicorn.error"}
{"event":"Booting worker with pid: 2733","level":"info","logger":"gunicorn.error"}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.