goauthentik / goauthentik/helm
Worker (Rust arbiter) shuts down silently within ~1ms of starting — no error, no warning — on Kubernetes/Helm, 2026.5.6
- Dominant language
- Mustache
- Stars
- 183
- Forks
- 69
- Avg merge
- 14h 21m
- Merged PRs (30d)
- 8
Description
After upgrading from 2026.2.3 to 2026.5.6 (Helm chart on Kubernetes), the authentik-worker pod starts, initializes normally, then has its worker subsystem shut down internally within roughly 1 millisecond of starting worker being logged — with no error, warning, or explanatory log line anywhere, even at trace level. The container itself keeps running afterward (health/metrics endpoints on the Rust side keep responding 200 OK indefinitely), so Kubernetes reports the pod as 1/1 Running with 0 restarts — but no scheduled or manually-triggered tasks are ever picked up (LDAP sync, blueprint apply, etc. all sit in "Waiting to run" forever).
To Reproduce
Deploy authentik 2026.5.6 via the official Helm chart on Kubernetes
Start the authentik-worker deployment (fresh single-pod start, no rolling overlap)
Observe worker logs — see below
Expected behavior
Worker starts and begins consuming queued tasks.
Actual behavior
Worker's internal task-processing subsystem shuts itself down almost instantly after starting, while the Rust-owned health/metrics server keeps running independently, masking the failure from Kubernetes' liveness probe.
Environment
authentik version: 2026.5.6
Deployment method: Helm chart (authentik-2026.5.6)
Platform: Kubernetes
Python: 3.14 (per logs)
Upgraded from: 2026.2.3
What I've ruled out
Not a rolling-update / old-pod-overlap race — reproduced with a clean scale --replicas=0 → scale --replicas=1 cycle, single pod, no old pod present
Not a CPU/resource-limit issue — reproduced identically with worker CPU limit raised from 500m to 2 cores, memory 1Gi
No preStop/postStart lifecycle hooks or custom probes configured
ps not available in the container to inspect processes directly (minimal image)
Relevant log (full worker startup, single clean pod, unedited):
```
{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1785759566.5264056, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1785759566.5270188, "count": 27}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5272782}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5675104}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5676851}
{"event": "Bootstrap completed", "level": "info", "logger": "bootstrap"}
{"event": "Not running as root, disabling permission fixes", "level": "info", "logger": "bootstrap"}
{"filename":"src/main.rs","level":"info","line_number":45,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.428886","event":"authentik is starting","version":"2026.5.6"}
{"filename":"src/main.rs","level":"trace","line_number":58,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.429133","event":"installing error formatting"}
{"filename":"src/main.rs","level":"trace","line_number":63,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.429189","event":"initializing Python"}
{"filename":"src/main.rs","level":"trace","line_number":65,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451135","event":"Python initialized"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":192,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451159","event":"loading config"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":195,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451850","event":"config loaded"}
{"filename":"packages/ak-common/src/tls.rs","level":"trace","line_number":41,"pid":7,"target":"authentik_common::tls","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451863","event":"installing rustls crypto provider"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":274,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.452387","event":"starting config file watcher"}
{"filename":"src/metrics/mod.rs","level":"info","line_number":36,"pid":7,"target":"authentik::metrics","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.452617","event":"installing Prometheus recorder"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":44,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.452670","event":"starting signals watcher"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":237,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.452942","event":"config file watcher started","paths":"[]"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":70,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.453474","event":"initializing database pool"}
{"filename":"src/metrics/mod.rs","level":"info","line_number":49,"pid":7,"target":"authentik::metrics","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.453500","event":"starting metrics upkeep runner"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":77,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.453562","addr":"\"/dev/shm/authentik-metrics.sock\" (pathname)","event":"starting metrics server"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":25,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.453624","addr":"[::]:9300","event":"starting metrics server"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":110,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.465616","event":"database pool initialized"}
{"filename":"src/worker/mod.rs","level":"info","line_number":65,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.465639","event":"starting worker","worker_id":1000}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":53,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.465871","event":"starting database watcher for config changes"}
{"filename":"src/worker/mod.rs","level":"info","line_number":286,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466104","event":"starting worker watcher"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":257,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466118","event":"arbiter has been told to shutdown gracefully"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":270,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466153","event":"all webservers have been shutdown, shutting down the other tasks gracefully"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":25,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.466143","addr":"[::]:9000","event":"starting worker server"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":239,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466164","event":"arbiter has been told to shutdown immediately"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":250,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466192","event":"all webservers have been shutdown, shutting down the other tasks immediately"}
{"filename":"src/worker/mod.rs","level":"info","line_number":104,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466256","event":"gracefully shutting down worker"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":80,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466498","event":"stopping signals watcher"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":265,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466559","event":"stopping config file watcher"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":62,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466633","event":"stopping database watcher for config changes"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":77,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466674","addr":"\"/dev/shm/authentik.sock\" (pathname)","event":"starting worker server"}
{"filename":"packages/ak-axum/src/tracing.rs","level":"info","line_number":48,"pid":7,"spans":[{"host":"localhost","http_headers":"{\"user_agent\": \"goauthentik.io/healthcheck\"}","method":"HEAD","name":"request","path":"/-/health/live/","remote":"::1","scheme":"http"}],"target":"authentik_axum::tracing","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:20:02.451842","event":"/-/health/live/","runtime":"0","status":200,"host":"localhost","http_headers":"{\"user_agent\": \"goauthentik.io/healthcheck\"}","method":"HEAD","name":"request","path":"/-/health/live/","remote":"::1","scheme":"http"}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the clean Helm deployment and compare the shutdown sequence in packages/ak-common/src/arbiter.rs with worker startup and shutdown in src/worker/mod.rs; also inspect the server events in packages/ak-axum/src/server.rs. Determine what triggers the arbiter shutdown and ensure the worker remains active, consumes queued tasks, and records an explanatory log if shutdown is intentional or caused by an error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, rust
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100