goauthentik / goauthentik/authentik

2026.8.1 server memory: glibc arena retention plus a ~1.7 GiB transient spike when browsing the admin UI (OOM)

Open
#25,683 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
25.6k
Forks
2k
Avg merge
1d 2h
Merged PRs (30d)
659

Description

### Describe the bug

On 2026.8.1 the `server` (web) process grows and OOMs in a way it did not on 2026.5.x. Two distinct behaviours, both reproducible here:

**1. glibc arena retention (a real leak).** A gunicorn web worker that handled one burst keeps the memory afterwards. An idle worker, serving no traffic, sat at:

```
Rss: 2110844 kB
Private_Dirty: 2002740 kB
Anonymous: 2090156 kB
threads: 8
```

That is 2 GiB of anonymous private-dirty held flat by a worker doing nothing. The image runs on Debian trixie / glibc 2.41 and sets no `MALLOC_*` tuning. The `DjangoUvicornWorker` runs blocking DB work in an 8-thread `ThreadPoolExecutor`; glibc opens up to `8 * ncpu` per-thread malloc arenas and never returns the freed heap to the OS, so worker RSS ratchets up and stays.

Setting `MALLOC_ARENA_MAX=2` and `MALLOC_TRIM_THRESHOLD_=131072` on the server (and worker) cut the post-burst retention from ~2 GiB to ~640 MiB and dropped idle baseline from ~850 MiB to ~480 MiB. That points at arena fragmentation, not live objects.

**2. A ~1.7 GiB transient spike when browsing the admin UI.** With arena tuning already in place, opening and navigating the admin interface (sources / providers, plus the events and user list views at `page_size=100`+) pushed the server working-set from a flat 709 MiB to 2677 MiB in about 90 seconds, crossing the memory limit and OOMKilling the process. 30-second Prometheus samples across the event:

```
15:07:00 709 MiB
15:08:00 709 MiB
15:09:00 2420 MiB
15:09:30 2625 MiB
15:10:00 2677 MiB <- OOMKilled here
15:10:30 826 MiB (after restart)
```

The kernel confirms the worker is the target:

```
Memory cgroup out of memory: Killed process (gunicorn: worker)
total-vm:2647228kB, anon-rss:1861000kB, file-rss:12916kB
Killed process (ThreadPoolExecutor) total-vm:2647228kB, anon-rss:1861000kB
```

The `ThreadPoolExecutor` thread and the gunicorn worker share the same `total-vm`, so the allocation lands in the async worker's thread pool.

Individual admin endpoints are cheap in isolation. `tracemalloc` around each of the ~11 admin-overview API calls peaks at ≤18 MiB, and 180 concurrent async calls to those endpoints only reached ~1360 MiB. The first hit of `/api/v3/events/events/?page_size=100` costs ~100 MiB (loading the 63 MB GeoLite2-City reader plus GeoIP/ASN context deserialization), then goes cheap once warm. I could not pin the single call that allocates the 1.7 GiB, so the trigger is a specific heavier admin action rather than the overview fan-out.

### To reproduce

1. Run 2026.8.1 server via the Helm chart, embedded outpost enabled, a real dataset behind it (here: 25 proxy providers, 38 OAuth2 providers, 39 applications, 1 LDAP source, GeoIP + ASN context processors on).
2. Watch server working-set / per-worker RSS.
3. Log in to the admin interface and browse: sources, providers, the events log and user list at large page sizes.
4. A worker climbs into the low GiB and OOMs if the limit is under ~3 GiB. Separately, leave a worker idle after a burst and watch it hold ~2 GiB of `Private_Dirty` until recycled.

### Expected behavior

Server RSS returns to baseline after a burst, and a single admin session does not allocate over a gigabyte in one session. On 2026.5.x this deployment ran at a 600–900 MiB daily max with 0–1 OOMs/day; on 2026.8.1 under the same load it spiked past 2.6 GiB and OOM-looped.

### Version and Deployment

- authentik 2026.8.1, `ghcr.io/goauthentik/server:2026.8.1`
- Helm chart, Kubernetes
- Python 3.14.7, gunicorn 26.0.0, `DjangoUvicornWorker`, `web.workers=2`, `web.threads=4`
- Debian trixie, glibc 2.41, 6 CPU node, no `MALLOC_*` env by default
- Postgres external (CNPG), embedded proxy outpost with 26 providers

### Notes

- The glibc `MALLOC_ARENA_MAX=2` finding may help #25470, where the worker leak has been hard to reproduce; arena retention would explain a slow climb that survives idle periods.
- Related: #25470 (worker memory leak in 2026.8.0), #25248 (proxy outpost sessions in /dev/shm), #25572 (LDAP sync cache errors).
- Happy to run memory dumps, `tracemalloc` snapshots, or `MALLOC_STATS` on request. `MALLOC_ARENA_MAX=2` is a workable mitigation for the retention half but does nothing for the transient spike.

Contributor guide

Open the contributing guide

Research direction

Start at the DjangoUvicornWorker and its ThreadPoolExecutor, then reproduce the issue while browsing the listed admin views and monitoring worker RSS. Investigate the glibc arena-retention behavior separately from the transient spike and compare against the reported 2026.5.x baseline. Done means the worker returns near baseline after bursts and an admin session does not trigger a gigabyte-scale allocation or OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, helm, kubernetes, postgresql, python
Domain
backend, infrastructure, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.