randomparity / randomparity/kdive

An out-of-band restart of a worker unit wedges the slot with no recovery path

Open
#2,481 1 comment 0 reactions 0 assignees View on GitHub
area:core-platform effort:M priority:P1 type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

If `kdive-live-worker@N.service` is stopped and started outside the lifecycle contract, the slot wedges permanently. Every subsequent lifecycle operation — including `stop` — returns `conflict / operator_recovery`, the `diagnostics` escape hatch returns an empty payload, `operator_recovery` is documented nowhere, and reinstalling the contract does not clear it. Recovery required hand-editing root-owned state on disk **and** an `UPDATE` against the database.

Found on a provisioned Ubuntu 26.04.1 KVM host while running the live tiers during verification of #2478. Present on `main`; #2478 does not touch this code.

## How it was triggered

Not by anything kdive did. Ubuntu's post-apt `needrestart` swept the service manager:

```
systemd[1]: Received SIGRTMIN+25 from PID (kill)
systemd[1]: Reexecuting.
systemd[1]: Stopping kdive-live-worker@1.service - KDIVE retained live worker slot 1...
systemd[1]: Stopped kdive-live-worker@1.service
systemd[1]: Started kdive-live-worker@1.service
kdive-live-worker-gate[]: kdive worker gate slot 1: release marker binding invariant failed
systemd[1]: kdive-live-worker@1.service: Main process exited, code=exited, status=1/FAILURE
```

The unit has `Restart=no`, so this was an external stop/start, not a restart loop. Any `systemctl daemon-reexec`, package-manager service sweep, or manual `systemctl restart` reaches the same state. An unattended-upgrades run on a long-lived runner would too.

## Why the restart cannot succeed

`deploy/systemd/bin/kdive-live-worker-gate` binds the release marker to systemd's `INVOCATION_ID`:

```python
expected = f"{generation}\n{invocation}\n".encode("ascii")
if marker != expected:
_fail(slot, "release marker binding invariant failed")
```

A restart mints a new `INVOCATION_ID`, but the on-disk marker still holds the previous one, so the gate exits 1 immediately. Observed on the host:

- retained `state.json`: `phase: "started"`, `invocation_id: dfef1e4d…/94b875d1…`
- release marker: `dfef1e4d…\n94b875d1…\n`
- restarted unit: `InvocationID=e124fd81…`, `ActiveState=failed`, `Result=exit-code`

Nothing rewrites the marker on a restart, so the slot can never come back on its own.

## Why it cannot be recovered through the contract

1. The `worker_incarnations` row stays `state=active, terminated_at=NULL` — the process died without publishing termination evidence, so it still holds the fence and blocks re-registration.
2. From then on **every** `scripts/live-stack/worker-lifecycle.sh` subcommand returns the same thing, `stop` included:

```json
{"ok":false,"code":"conflict","message":"retained lifecycle facts conflict",
"retry_action":"operator_recovery",
"slots":[{"slot":1,"unit":"kdive-live-worker@1.service","phase":"started","code":"conflict"}]}
```

The script's interface is `start COUNT|status|stop|diagnostics`. None of them clears the state, so the contract cannot recover itself.
3. `diagnostics`, the designated escape hatch, withholds the payload:

```json
{"ok":false,"code":"diagnostics_withheld","message":"diagnostics withheld for one or more slots",
"retry_action":"operator_recovery","diagnostics":""}
```
4. `operator_recovery` appears nowhere in `docs/` or `deploy/` — only in `src/kdive/processes/lifecycle/systemd/` and its tests. No runbook says what it means or what an operator should do.
5. Re-running `deploy/systemd/install-live-worker-lifecycle.sh` does not reset retained slot facts.

## The error messages actively hide the cause

`src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py:713` collapses three distinct exceptions into one opaque string:

```python
if isinstance(error, (LifecycleConflict, StateConflict, SystemdConflict)):
return "conflict", "operator_recovery", "retained lifecycle facts conflict"
```

and `_register` wraps *every* exception as an availability problem:

```python
except Exception as exc:
raise _AuthorityUnavailable("worker registration authority unavailable") from exc
```

which surfaces as `dependency_unavailable / restore_database / "database authority is unavailable"`. During recovery this reported a database outage while PostgreSQL was healthy and logging no errors at all; the real cause was elsewhere entirely. Chasing a misattributed message cost most of the recovery time.

## What recovery actually required

None of this is documented:

1. `systemctl stop` + `systemctl reset-failed kdive-live-worker@1.service`
2. move aside root-owned `state.json` and `release` under `/var/lib/kdive/live-workers/slots/1/`
3. `UPDATE worker_incarnations SET state='terminated', terminated_at=now(), outcome='failed' WHERE state='active'`
4. re-run the bring-up

Retained state spans the filesystem *and* the database, and there is no tool that reconciles the two.

## Suggested scope

- Make the gate tolerate a legitimate restart: re-derive or re-publish the release marker for the current `INVOCATION_ID` rather than failing closed on a value that is guaranteed to change.
- Give `worker-lifecycle.sh` a `recover` (or `stop --force`) subcommand that retires a slot whose unit is dead, clearing both the on-disk facts and the `worker_incarnations` fence in one operation.
- Preserve the discriminating cause in the response (which invariant, which conflict class) instead of one flattened string, and stop mapping arbitrary registration exceptions onto "database authority is unavailable".
- Document `operator_recovery` in the lifecycle README with the procedure.

Contributor guide

Open the contributing guide

Research direction

Start with deploy/systemd/bin/kdive-live-worker-gate and src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py, including their tests, then inspect scripts/live-stack/worker-lifecycle.sh. Trace restart handling, retained filesystem facts, and the worker_incarnations fence before choosing a recovery design. Done means legitimate restarts recover, operator recovery reconciles both stores, causes remain distinguishable, and the lifecycle README documents operator_recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, postgresql, python, shell, ubuntu
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.