app-server daemon: auto-update force-kills active turns after a 60s drain budget, and cannot be disabled

app-server daemon: auto-update force-kills active turns after a 60s drain budget, and cannot be disabled

Summary

On a long-lived codex remote-control daemon, the auto-updater restarts the app-server as soon as a new release lands. The app-server implements a correct graceful drain that waits for running assistant turns — but the daemon only grants it 60 seconds before sending SIGKILL. For any workload where a turn routinely lasts minutes (agents running build/test suites), the drain can essentially never complete, so every auto-update destroys in-flight work across every thread on the host.

There is also no supported way to turn auto-update off, so an operator cannot opt out of the behavior.

Environment

• codex-cli / app-server 0.149.0 → 0.150.0 (standalone managed install), Debian 13 x86_64
• daemon started via codex remote-control start; multiple TUI clients attached over an SSH-forwarded unix socket
• workload: 13 long-lived threads; typical turn runs a Go build/test suite lasting 3–30 minutes

What happened

2026-08-26, unattended:

• 20:16 — updater installed 0.150.0 (symlink ~/.codex/packages/standalone/current re-pointed).
• 20:16 → 20:17:51 — app-server restarted; every attached client connection dropped simultaneously; turns that were mid-execution were terminated. The ~90s gap matches the grace period elapsing before the forced kill.
• Recovery required reconnecting each client and interrupting/continuing each thread; turns that were awaiting an exec at kill time exhibited the hang described in #40306.

Mechanism (source refs, rust-v0.150.0)

The graceful path exists and is well-implemented:

• app-server/src/lib.rs:250-253 — on SIGTERM the server logs "received shutdown signal; entering graceful restart drain (connections=…, runningAssistantTurns=…, requests still accepted until no assistant turns are running)".
• app-server/src/lib.rs:257-262 — update() returns ShutdownAction::Finish only when self.forced || running_turn_count == 0, and otherwise logs "waiting for N running assistant turn(s) to finish".

The daemon then overrides it:

• app-server-daemon/src/backend/pid.rs:520 — stop() sends SIGTERM.
• app-server-daemon/src/backend/pid.rs:23-24 — STOP_GRACE_PERIOD = 60s, STOP_TIMEOUT = 70s.
• app-server-daemon/src/backend/pid.rs:253-277 — after STOP_GRACE_PERIOD elapses, force_terminate_process(pid) → SIGKILL (pid.rs:535).

Restart eligibility is version-only and has no turn awareness:

• app-server-daemon/src/lib.rs:804-818 — restart_decision compares info.app_server_version against the managed version.
• app-server-daemon/src/lib.rs:363-372 — the RestartIfRunningOutcome::Busy branch reflects the daemon operation lock, not running turns.

Auto-update cannot be disabled:

• app-server-daemon/src/settings.rs:11-13 — DaemonSettings has exactly one field, remote_control_enabled; settings.json cannot express an update preference.
• app-server-daemon/src/lib.rs:608-612 — bootstrap unconditionally starts pid_update_loop_backend; there is no gate.
• app-server-daemon/src/lib.rs:619 — auto_update_enabled: true in the status payload is a hardcoded literal, not a setting, which makes the reported field misleading.

No CLI flag or environment variable appears to control it. The only working workaround is to kill the codex app-server daemon pid-update-loop process after every daemon start (it does not respawn on its own, and killing it does not disturb the running app-server), or to bypass codex app-server daemon entirely and supervise codex app-server directly.

Requests

1. Make the drain budget configurable, and allow it to be unbounded. A server operator should be able to say "never SIGKILL an app-server that is still draining turns" (or set a budget in the tens of minutes). The 60s default is sensible for an interactive desktop session and unusable for turns that run build/test suites. The forced path should remain available explicitly (e.g. a second signal / --force), as on_signal already models.
2. Provide a supported way to disable auto-update — a DaemonSettings field plus codex app-server daemon disable-auto-update (mirroring the existing enable-remote-control / disable-remote-control pair) would fit the current surface. Operators running unattended fleets need updates to be a deliberate, scheduled action.
3. Optionally: defer rather than kill. Since the updater already loops with RESTART_RETRY_INTERVAL on Busy, treating "turns are running" as a retryable condition — rather than starting a drain that will be killed — would make auto-update safe by default without any configuration.

Also worth fixing regardless: autoUpdateEnabled in the status output currently reports a constant, so tooling cannot detect the actual state.

Related

• #40306 — turns awaiting an exec hang indefinitely when their client connection dies; the forced kill in this issue is one way to trigger that state.

Start with app-server-daemon/src/backend/pid.rs for the stop timeout and app-server-daemon/src/lib.rs for restart decisions, bootstrap, and status reporting. Then read app-server-daemon/src/settings.rs and the existing enable/disable remote-control commands to understand the settings surface. Done means the requested drain and auto-update behavior is configurable, disabling is supported, and the reported auto-update state reflects the actual setting.