Worker leaks ~1000 unconnected AF_UNIX sockets and hits RLIMIT_NOFILE=1024 after ~41h, going permanently unhealthy (2026.8.0, docker-compose)

> **Disclosure:** this report — investigation, data collection and wording — was produced by an AI assistant (Claude) driven by the server operator, from read-only inspection of a live production host. Every number below comes from a command whose output I still have. Nothing is inferred where it says "confirmed", and the section marked *Hypothesis* is explicitly not proven.

### Describe the bug

The `worker` container's Python process (`python -m lifecycle.worker_process 1000 /dev/shm/authentik-worker-1000.sock`) slowly leaks AF_UNIX sockets. After roughly 41 hours of idle-ish operation it reaches its soft `RLIMIT_NOFILE` of 1024 and can no longer create *any* file descriptor. From that point on:

- every DB access fails at DNS resolution: `failed to resolve host 'postgresql': [Errno 24] Too many open files`
- the Rust supervisor's liveness probe fails with `hyper_util::client::legacy::Error(Connect, Os { code: 11, kind: WouldBlock })`, and `/-/health/live/` returns 503
- the container is flagged `unhealthy` and never recovers; only a restart clears it

Nothing external is involved: no container on the host restarted (`RestartCount=0` on all three authentik containers), the host is nowhere near any fd limit, and `authentik-server` running the **same image** is stable at ~30 fds.

#### Confirmed evidence

Soft limit reached exactly:

```
$ PID=
$ ls /proc/$PID/fd | wc -l
1024
$ grep 'Max open files' /proc/$PID/limits
Max open files 1024 524288 files
```

fd breakdown of that process:

```
1003 socket:[N]
6 pipe:[N]
2 anon_inode:inotify
12 /dev/shm/authentik_prometheus_tmp/*.db
1 /dev/null
1 .
```

Resolving all 1003 socket fds against the container netns tables (`/proc/$PID/net/{unix,tcp,tcp6,udp,udp6}`, joined by inode):

```
AF_UNIX matches: 997
996 RefCount=00000002 Flags=00000000 Type=0001 St=01 Path=
1 RefCount=00000002 Flags=00010000 Type=0001 St=01 Path=/dev/shm/authentik-worker-1000.sock
AF_INET/tcp matches: 4
AF_INET/tcp6 matches: 0
AF_INET/udp matches: 0
AF_INET/udp6 matches: 0
```

So: **996 unnamed AF_UNIX SOCK_STREAM sockets**, plus the one legitimate listening socket. `Flags=0` (not listening) and `St=01` (`SS_UNCONNECTED`). Per `unix_seq_show()` in Linux v6.8 `net/unix/af_unix.c` (the kernel this host runs), a socket prints `St=03` (`SS_CONNECTED`) only when `sk_state == TCP_ESTABLISHED`, and a peer closing its end does **not** reset that field (`unix_release_sock()` sets `sk_shutdown`/`sk_err` on the peer, not `sk_state`). So these are sockets that were **created and never successfully connected**, then never closed. They are *not* leftover accepted connections.

Neither the Rust supervisor nor the server leaks:

```
worker container: server container:
1024 python -m lifecycle.worker_process 1000 ... 33 gunicorn: worker [authentik.root.asgi:application]
23 authentik worker (rust supervisor) 29 authentik server
0 dumb-init -- ak worker 27 gunicorn: worker [...]
12 gunicorn: master [...]
```

The host itself is idle w.r.t. fds:

```
/proc/sys/fs/file-nr : 9888 0 9223372036854775807
/proc/sys/fs/file-max: 9223372036854775807
```

#### Timeline

| when | what |
|---|---|
| `2026-08-21T18:54:56Z` | worker container started (`RestartCount=0`, never restarted since) |
| `2026-08-21T18:54:57Z` | `lifecycle.worker_process` starts |
| `2026-08-23T12:25:23Z` | **first** `Too many open files` in the logs — ~41h30 later |
| `2026-08-23T21:24Z` | still saturated: 12821 EMFILE lines logged, worker `unhealthy`, pegged at 1024 fds |

#### Hypothesis (NOT proven)

I could not identify the allocation site from `/proc` alone. What is established is that the leaked sockets are unconnected, client-side AF_UNIX sockets — something in the worker process repeatedly does `socket(AF_UNIX, SOCK_STREAM)` (or a `connect`/`accept` that never completes) and drops the fd without closing it. The only AF_UNIX endpoints in this container are `/dev/shm/authentik-worker-1000.sock`, `/dev/shm/authentik.sock` and `/dev/shm/authentik-metrics.sock`, so the IPC between the Rust supervisor and `lifecycle/worker_process.py` is the natural place to look — but I want to be explicit that I have **not** demonstrated which call site allocates them. Note that the "unconnected" state argues against the naive "the `UnixSocketServer` forgets to close accepted connections" reading.

I'm happy to run `strace -f -p -e trace=socket,connect,accept4,close -tt` or `py-spy dump` against the live process and attach the output — just say what you want captured. I also have a full read-only diagnostic bundle (environment, deployment, fd inventory, socket classification, host state, log excerpts, redacted compose) I can attach on request.

### How to reproduce

1. Run 2026.8.0 with docker-compose.
2. Leave the worker running. No special load is required — this is a small single-tenant install whose only recurring worker activity is the built-in periodic tasks.
3. Watch the fd count of the Python worker process:

```sh
PID=$(docker top authentik-worker-1 -eo pid | tail -n +2 | while read p; do
tr -d '\0' < /proc/$p/cmdline | grep -q lifecycle.worker_process && echo $p; done)
watch -n 60 "ls /proc/$PID/fd | wc -l"
```

4. The count climbs monotonically and never comes back down. It reached 1024 in ~41h30 here, at which point the worker goes `unhealthy` permanently.

### Expected behavior

The worker's file descriptor count stays bounded, and the container does not need periodic restarts to stay healthy.

### Additional context

- The 1024 soft limit is arguably a second, independent issue. The `worker` service declares no `ulimits:`, and the daemon has no `default-ulimits` in `/etc/docker/daemon.json`, so the container inherits Docker's default `nofile` of **1024 soft / 524288 hard** — confirmed in `/proc/$PID/limits` for all three PIDs in the container (`dockerd` itself runs at 524287/524288, so this is not inherited from the daemon). Setting `ulimits: {nofile: {soft: 65536, hard: 524288}}` on the worker turns a 41-hour outage cycle into a multi-month one — it only hides the leak, but it may still be worth raising the default in the reference compose file, since the worker is single-process and 1024 is very tight.
- Only the worker is affected. `authentik-server` from the same image, same host, same uptime, sits at 27-33 fds per process.
- The first EMFILE of the run surfaced as an outbound HTTPS failure rather than a DB one, which is consistent with plain fd exhaustion rather than a database problem:

```
2026-08-23T12:25:23.476538126Z {"event": "Failed to process message authentik.admin.tasks.update_latest_version() with unhandled exception.",
"exc_type": "SSLError",
"exc_value": "HTTPSConnectionPool(host='version.goauthentik.io', port=443): Max retries exceeded with url: /version.json (Caused by SSLError(OSError(24, 'Too many open files')))"}
```

- Environment: Ubuntu 24.04.4 LTS, kernel 6.8.0-137-generic, x86_64. Docker 29.7.2, overlay2, cgroup v2 (systemd), runc. Image `ghcr.io/goauthentik/server@sha256:7421753cfea67e89a6d295a1f0173ccea3866b33768c88dad90453b151cdcfd5` (built 2026-08-18), Python 3.14 inside. 3 services: `postgresql` (postgres:16-alpine), `server`, `worker`. No Redis, embedded outpost only, Caddy in front of `authentik-server:9000`.

### Deployment Method

Docker

### Version

2026.8.0

### Relevant log output

```shell
warning | event=Failed to post_process_message(authentik.admin.tasks.update_latest_version()) due to a connection error: failed to resolve host 'postgresql': [Errno 24] Too many open files
The operation will be retried in 5 seconds until the connection recovers.
If you restart this worker before this operation succeeds, the message will be re-processed later. logger=dramatiq.worker.ConsumerThread(default) timestamp=2026-08-23T12:46:55.558843

warning | event=Failed to post_process_message(authentik.sources.oauth.tasks.update_well_known_jwks()) due to a connection error: failed to resolve host 'postgresql': [Errno 24] Too many open files
The operation will be retried in 5 seconds until the connection recovers.
If you restart this worker before this operation succeeds, the message will be re-processed later. logger=dramatiq.worker.ConsumerThread(default) timestamp=2026-08-23T12:46:56.073402

WRN | filename=src/worker/mod.rs line_number=151 pid=7 spans=[{"host":"localhost","http_headers":"{\"user_agent\": \"goauthentik.io/healthcheck\"}","method":"GET","name":"request","path":"/-/health/live/","remote":"::1","scheme":"http"},{"name":"health_live"},{"name":"health_live"},{"name":"health_live","worker_id":1000}] target=authentik::worker thread_id=ThreadId(3) thread_name=tokio-0 timestamp=2026-08-23T12:46:58.090747 err=hyper_util::client::legacy::Error(Connect, Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) event=failed to send health live request to worker name=health_live worker_id=1000

WRN | filename=src/worker/healthcheck.rs line_number=54 pid=7 target=authentik::worker::healthcheck thread_id=ThreadId(3) thread_name=tokio-0 timestamp=2026-08-23T12:46:58.090837 err=
0: client error (Connect)
1: Resource temporarily unavailable (os error 11)

Location:
src/worker/mod.rs:147

---------------------------------- SPANTRACE ----------------------------------

0: authentik::worker::health_live with worker_id=1000
at src/worker/mod.rs:139
1: authentik::worker::health_live
at src/worker/mod.rs:259
2: authentik::worker::healthcheck::health_live
at src/worker/healthcheck.rs:40
3: authentik_axum::tracing::request with path=/-/health/live/ method=GET http_headers={"user_agent": "goauthentik.io/healthcheck"}
at packages/ak-axum/src/tracing.rs:26
4: tokio::task::runtime.spawn with kind=task task.id=64144 loc.file="axum-server-0.8.0/src/server.rs" loc.line=308

event=failed to check workers health liveness name=health_live

INF | filename=packages/ak-axum/src/tracing.rs line_number=48 pid=7 target=authentik_axum::tracing thread_id=ThreadId(3) thread_name=tokio-0 timestamp=2026-08-23T12:46:58.090973 event=/-/health/live/ runtime=0 status=503 host=localhost method=GET name=request path=/-/health/live/ remote=::1 scheme=http
```

Start with lifecycle/worker_process.py and the worker IPC paths, using the offered strace command to identify which socket allocations lack cleanup. Check the related supervisor and health-check entries in src/worker/mod.rs and src/worker/healthcheck.rs. Done means the worker's descriptor count stays bounded during reproduction and it remains healthy without restart.

Written by the indexing model from the issue text.