randomparity / randomparity/kdive
An out-of-band restart of a worker unit wedges the slot with no recovery path
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
If `kdive-live-worker@N.service` is stopped and started outside the lifecycle contract, the slot wedges permanently. Every subsequent lifecycle operation — including `stop` — returns `conflict / operator_recovery`, the `diagnostics` escape hatch returns an empty payload, `operator_recovery` is documented nowhere, and reinstalling the contract does not clear it. Recovery required hand-editing root-owned state on disk **and** an `UPDATE` against the database.
Found on a provisioned Ubuntu 26.04.1 KVM host while running the live tiers during verification of #2478. Present on `main`; #2478 does not touch this code.
## How it was triggered
Not by anything kdive did. Ubuntu's post-apt `needrestart` swept the service manager:
```
systemd[1]: Received SIGRTMIN+25 from PID (kill)
systemd[1]: Reexecuting.
systemd[1]: Stopping kdive-live-worker@1.service - KDIVE retained live worker slot 1...
systemd[1]: Stopped kdive-live-worker@1.service
systemd[1]: Started kdive-live-worker@1.service
kdive-live-worker-gate[]: kdive worker gate slot 1: release marker binding invariant failed
systemd[1]: kdive-live-worker@1.service: Main process exited, code=exited, status=1/FAILURE
```
The unit has `Restart=no`, so this was an external stop/start, not a restart loop. Any `systemctl daemon-reexec`, package-manager service sweep, or manual `systemctl restart` reaches the same state. An unattended-upgrades run on a long-lived runner would too.
## Why the restart cannot succeed
`deploy/systemd/bin/kdive-live-worker-gate` binds the release marker to systemd's `INVOCATION_ID`:
```python
expected = f"{generation}\n{invocation}\n".encode("ascii")
if marker != expected:
_fail(slot, "release marker binding invariant failed")
```
A restart mints a new `INVOCATION_ID`, but the on-disk marker still holds the previous one, so the gate exits 1 immediately. Observed on the host:
- retained `state.json`: `phase: "started"`, `invocation_id: dfef1e4d…/94b875d1…`
- release marker: `dfef1e4d…\n94b875d1…\n`
- restarted unit: `InvocationID=e124fd81…`, `ActiveState=failed`, `Result=exit-code`
Nothing rewrites the marker on a restart, so the slot can never come back on its own.
## Why it cannot be recovered through the contract
1. The `worker_incarnations` row stays `state=active, terminated_at=NULL` — the process died without publishing termination evidence, so it still holds the fence and blocks re-registration.
2. From then on **every** `scripts/live-stack/worker-lifecycle.sh` subcommand returns the same thing, `stop` included:
```json
{"ok":false,"code":"conflict","message":"retained lifecycle facts conflict",
"retry_action":"operator_recovery",
"slots":[{"slot":1,"unit":"kdive-live-worker@1.service","phase":"started","code":"conflict"}]}
```
The script's interface is `start COUNT|status|stop|diagnostics`. None of them clears the state, so the contract cannot recover itself.
3. `diagnostics`, the designated escape hatch, withholds the payload:
```json
{"ok":false,"code":"diagnostics_withheld","message":"diagnostics withheld for one or more slots",
"retry_action":"operator_recovery","diagnostics":""}
```
4. `operator_recovery` appears nowhere in `docs/` or `deploy/` — only in `src/kdive/processes/lifecycle/systemd/` and its tests. No runbook says what it means or what an operator should do.
5. Re-running `deploy/systemd/install-live-worker-lifecycle.sh` does not reset retained slot facts.
## The error messages actively hide the cause
`src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py:713` collapses three distinct exceptions into one opaque string:
```python
if isinstance(error, (LifecycleConflict, StateConflict, SystemdConflict)):
return "conflict", "operator_recovery", "retained lifecycle facts conflict"
```
and `_register` wraps *every* exception as an availability problem:
```python
except Exception as exc:
raise _AuthorityUnavailable("worker registration authority unavailable") from exc
```
which surfaces as `dependency_unavailable / restore_database / "database authority is unavailable"`. During recovery this reported a database outage while PostgreSQL was healthy and logging no errors at all; the real cause was elsewhere entirely. Chasing a misattributed message cost most of the recovery time.
## What recovery actually required
None of this is documented:
1. `systemctl stop` + `systemctl reset-failed kdive-live-worker@1.service`
2. move aside root-owned `state.json` and `release` under `/var/lib/kdive/live-workers/slots/1/`
3. `UPDATE worker_incarnations SET state='terminated', terminated_at=now(), outcome='failed' WHERE state='active'`
4. re-run the bring-up
Retained state spans the filesystem *and* the database, and there is no tool that reconciles the two.
## Suggested scope
- Make the gate tolerate a legitimate restart: re-derive or re-publish the release marker for the current `INVOCATION_ID` rather than failing closed on a value that is guaranteed to change.
- Give `worker-lifecycle.sh` a `recover` (or `stop --force`) subcommand that retires a slot whose unit is dead, clearing both the on-disk facts and the `worker_incarnations` fence in one operation.
- Preserve the discriminating cause in the response (which invariant, which conflict class) instead of one flattened string, and stop mapping arbitrary registration exceptions onto "database authority is unavailable".
- Document `operator_recovery` in the lifecycle README with the procedure.
Contributor guide
Research direction
Start with deploy/systemd/bin/kdive-live-worker-gate and src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py, including their tests, then inspect scripts/live-stack/worker-lifecycle.sh. Trace restart handling, retained filesystem facts, and the worker_incarnations fence before choosing a recovery design. Done means legitimate restarts recover, operator recovery reconciles both stores, causes remain distinguishable, and the lifecycle README documents operator_recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, postgresql, python, shell, ubuntu
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100