goauthentik / goauthentik/helm

Worker (Rust arbiter) shuts down silently within ~1ms of starting — no error, no warning — on Kubernetes/Helm, 2026.5.6

Open
#503 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Mustache
Stars
183
Forks
69
Avg merge
14h 21m
Merged PRs (30d)
8

Description

After upgrading from 2026.2.3 to 2026.5.6 (Helm chart on Kubernetes), the authentik-worker pod starts, initializes normally, then has its worker subsystem shut down internally within roughly 1 millisecond of starting worker being logged — with no error, warning, or explanatory log line anywhere, even at trace level. The container itself keeps running afterward (health/metrics endpoints on the Rust side keep responding 200 OK indefinitely), so Kubernetes reports the pod as 1/1 Running with 0 restarts — but no scheduled or manually-triggered tasks are ever picked up (LDAP sync, blueprint apply, etc. all sit in "Waiting to run" forever).

To Reproduce

Deploy authentik 2026.5.6 via the official Helm chart on Kubernetes
Start the authentik-worker deployment (fresh single-pod start, no rolling overlap)
Observe worker logs — see below

Expected behavior

Worker starts and begins consuming queued tasks.

Actual behavior

Worker's internal task-processing subsystem shuts itself down almost instantly after starting, while the Rust-owned health/metrics server keeps running independently, masking the failure from Kubernetes' liveness probe.

Environment

authentik version: 2026.5.6
Deployment method: Helm chart (authentik-2026.5.6)
Platform: Kubernetes
Python: 3.14 (per logs)
Upgraded from: 2026.2.3

What I've ruled out

Not a rolling-update / old-pod-overlap race — reproduced with a clean scale --replicas=0 → scale --replicas=1 cycle, single pod, no old pod present
Not a CPU/resource-limit issue — reproduced identically with worker CPU limit raised from 500m to 2 cores, memory 1Gi
No preStop/postStart lifecycle hooks or custom probes configured
ps not available in the container to inspect processes directly (minimal image)

Relevant log (full worker startup, single clean pod, unedited):

```
{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1785759566.5264056, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1785759566.5270188, "count": 27}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5272782}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5675104}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1785759566.5676851}
{"event": "Bootstrap completed", "level": "info", "logger": "bootstrap"}
{"event": "Not running as root, disabling permission fixes", "level": "info", "logger": "bootstrap"}
{"filename":"src/main.rs","level":"info","line_number":45,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.428886","event":"authentik is starting","version":"2026.5.6"}
{"filename":"src/main.rs","level":"trace","line_number":58,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.429133","event":"installing error formatting"}
{"filename":"src/main.rs","level":"trace","line_number":63,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.429189","event":"initializing Python"}
{"filename":"src/main.rs","level":"trace","line_number":65,"pid":7,"target":"authentik","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451135","event":"Python initialized"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":192,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451159","event":"loading config"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":195,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451850","event":"config loaded"}
{"filename":"packages/ak-common/src/tls.rs","level":"trace","line_number":41,"pid":7,"target":"authentik_common::tls","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.451863","event":"installing rustls crypto provider"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":274,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.452387","event":"starting config file watcher"}
{"filename":"src/metrics/mod.rs","level":"info","line_number":36,"pid":7,"target":"authentik::metrics","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.452617","event":"installing Prometheus recorder"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":44,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.452670","event":"starting signals watcher"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":237,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.452942","event":"config file watcher started","paths":"[]"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":70,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.453474","event":"initializing database pool"}
{"filename":"src/metrics/mod.rs","level":"info","line_number":49,"pid":7,"target":"authentik::metrics","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.453500","event":"starting metrics upkeep runner"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":77,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.453562","addr":"\"/dev/shm/authentik-metrics.sock\" (pathname)","event":"starting metrics server"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":25,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.453624","addr":"[::]:9300","event":"starting metrics server"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":110,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.465616","event":"database pool initialized"}
{"filename":"src/worker/mod.rs","level":"info","line_number":65,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.465639","event":"starting worker","worker_id":1000}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":53,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.465871","event":"starting database watcher for config changes"}
{"filename":"src/worker/mod.rs","level":"info","line_number":286,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466104","event":"starting worker watcher"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":257,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466118","event":"arbiter has been told to shutdown gracefully"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":270,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466153","event":"all webservers have been shutdown, shutting down the other tasks gracefully"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":25,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(2)","thread_name":"tokio-0","timestamp":"2026-08-03T12:19:27.466143","addr":"[::]:9000","event":"starting worker server"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":239,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466164","event":"arbiter has been told to shutdown immediately"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":250,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(1)","thread_name":"main","timestamp":"2026-08-03T12:19:27.466192","event":"all webservers have been shutdown, shutting down the other tasks immediately"}
{"filename":"src/worker/mod.rs","level":"info","line_number":104,"pid":7,"target":"authentik::worker","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466256","event":"gracefully shutting down worker"}
{"filename":"packages/ak-common/src/arbiter.rs","level":"info","line_number":80,"pid":7,"target":"authentik_common::arbiter","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466498","event":"stopping signals watcher"}
{"filename":"packages/ak-common/src/config/mod.rs","level":"info","line_number":265,"pid":7,"target":"authentik_common::config","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466559","event":"stopping config file watcher"}
{"filename":"packages/ak-common/src/db.rs","level":"info","line_number":62,"pid":7,"target":"authentik_common::db","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466633","event":"stopping database watcher for config changes"}
{"filename":"packages/ak-axum/src/server.rs","level":"info","line_number":77,"pid":7,"target":"authentik_axum::server","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:19:27.466674","addr":"\"/dev/shm/authentik.sock\" (pathname)","event":"starting worker server"}
{"filename":"packages/ak-axum/src/tracing.rs","level":"info","line_number":48,"pid":7,"spans":[{"host":"localhost","http_headers":"{\"user_agent\": \"goauthentik.io/healthcheck\"}","method":"HEAD","name":"request","path":"/-/health/live/","remote":"::1","scheme":"http"}],"target":"authentik_axum::tracing","thread_id":"ThreadId(3)","thread_name":"tokio-1","timestamp":"2026-08-03T12:20:02.451842","event":"/-/health/live/","runtime":"0","status":200,"host":"localhost","http_headers":"{\"user_agent\": \"goauthentik.io/healthcheck\"}","method":"HEAD","name":"request","path":"/-/health/live/","remote":"::1","scheme":"http"}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the clean Helm deployment and compare the shutdown sequence in packages/ak-common/src/arbiter.rs with worker startup and shutdown in src/worker/mod.rs; also inspect the server events in packages/ak-axum/src/server.rs. Determine what triggers the arbiter shutdown and ensure the worker remains active, consumes queued tasks, and records an explanatory log if shutdown is intentional or caused by an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, rust
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.