Graceful shutdown exits before the workflow queue releases in-flight jobs — async turns wedged ~14 min after restart

Summary

On a self-hosted deployment, restarting the process while a turn is in flight leaves that turn's msg_ job locked in Postgres by a worker that no longer exists. Nothing retries it until eve's own :backstop: job fires roughly 14 minutes later. The turn is not lost — the complete answer eventually arrives — but for a chat agent the assistant is visibly dead for a quarter of an hour after every deploy.

Reproduced 3/3.

Environment

• eve 0.32.0, self-hosted (eve build then eve start), Node 24, Docker
• @workflow/world-postgres 5.0.0-beta.32 (createWorld() in agent/agent.ts)
• graphile-worker 0.16.6 (transitive)
• Postgres 16

Expected

A restart during an async turn releases the in-flight job so the new process can pick it up immediately.

Actual

The job row stays locked_by = <dead worker>, attempts = 1, indefinitely. Recovery waits for eve's step_…:backstop: job (~14 min) or graphile-worker's stale-lock reclaim (~4 h), whichever lands first.

Measured: restart at 15:23:29 → turn completed 15:38:03. 14m34s.

Reproduction

1. Self-host with @workflow/world-postgres as the world.
2. POST /eve/v1/session with a prompt long enough to run ~60s. It returns {"status":"accepted"} immediately — the work is asynchronous.
3. A few seconds later, restart the process (docker restart).
4. Query the queue:

select id, key, attempts, locked_by, locked_at
  from graphile_worker.jobs where key like 'msg_%';

The turn's job is still locked by the previous worker. The run row sits at status = running with its step count frozen, so polling the run tells you nothing.

Root cause

Two shutdown paths race, and the one that wins exits the process.

1. graphile-worker installs its own SIGTERM handler. world-postgres only passes noHandleSignals when applicationManagedShutdown is set, and that is off by default — so on SIGTERM graphile logs "attempting global graceful shutdown… (all termination signals will be ignored for the next 5 seconds)" and begins releasing its jobs via failJobs().

2. Meanwhile eve's server shutdown closes the HTTP server and counts down SERVER_SHUTDOWN_TIMEOUT (default 5s) waiting only for HTTP connections to drain. A turn is asynchronous — the POST returned long ago — so there are no connections to wait for. server.close() resolves on the first loop iteration and the process exits.

The result is that the process exits immediately, before graphile-worker reaches failJobs(). The lock survives a perfectly clean shutdown.

The tell is that docker stop returns in 0 seconds with exit code 0. This is not a SIGKILL at the end of a grace period — raising the container's stop grace period changes nothing, because the process is exiting voluntarily and instantly. (I chased that theory first; it is a dead end.)

Related: SERVER_SHUTDOWN_TIMEOUT does not help either, since the countdown loop breaks as soon as the server closes.

Workaround

graphile-worker's supported reclaim API, run after the restart:

select graphile_worker.force_unlock_workers(array['worker-…']);

Unlock only workers whose lock predates the current process start, or you risk running a live job twice. With a wrapper that restarts and then reclaims, recovery drops from 14m34s to 26s end to end. Three previously wedged turns — one stuck 2h51m — all resumed within 30s of that call.

Suggested fix

Have the shutdown path await the workflow world's close() before exiting. world-postgres already implements it (it calls runner.stop(), which is what reaches failJobs()); it just is not awaited on the way out. Waiting for HTTP connections is the wrong completion signal for a runtime whose actual work is asynchronous.

If the intended contract is instead that applications own this, then applicationManagedShutdown needs to be the documented default for self-hosting — but note that on its own it makes things worse, since it only stops graphile handling signals and start-production-server.js has no SIGTERM handler that closes the world.

Start at createWorld() in agent/agent.ts and inspect the shutdown path around SERVER_SHUTDOWN_TIMEOUT; compare it with start-production-server.js and the workflow world's close() behavior. Reproduce the async POST/restart sequence and inspect graphile_worker.jobs. Done means shutdown awaits world cleanup and the in-flight job is immediately recoverable after restart.