Epic: recover a live worker slot after an out-of-band unit restart

# PRD: recover a live worker slot after an out-of-band unit restart

Parent report: #2481.

## Problem

`kdive-live-worker@N.service` is stopped and started outside the lifecycle contract — by
`systemctl daemon-reexec`, a package-manager service sweep (`needrestart`), unattended
upgrades, or a manual `systemctl restart`. The slot then wedges permanently and no
subcommand of the shipped contract can clear it.

Three separate defects combine to produce that:

1. **The gate cannot survive a restart.** `_wait_for_release`
(`deploy/systemd/bin/kdive-live-worker-gate:118-136`) requires the root-owned release
marker to equal `f"{generation}\n{invocation}\n"` (`:133-135`), where `invocation` is
systemd's `INVOCATION_ID` for the gate's own process. A restart mints a new
`INVOCATION_ID` while the on-disk marker still holds the previous one, so the gate calls
`_fail` (`:50-52`, `SystemExit(1)`) with `release marker binding invariant failed`.
Nothing rewrites the marker on a restart, so the slot cannot come back on its own.

2. **No lifecycle operation can retire the wedged slot.** The worker died without
publishing termination evidence, so its `worker_incarnations` row stays
`state=active, terminated_at=NULL` and still holds the re-registration fence. In
`systemd_worker_lifecycle.py`, `_terminal_observation` (:654-671) raises
`LifecycleConflict("systemd invocation does not match retained state")` (:665) for the
restarted unit, and that reaches every caller — `stop` included. The shell client's
interface is `start COUNT|status|stop|diagnostics`
(`scripts/live-stack/worker-lifecycle.sh:17-19`), and none of those clears the state, so
the contract cannot recover itself. Recovery required hand-editing root-owned state under
the slot directory **and** an `UPDATE` against `worker_incarnations`.

3. **The responses misattribute the cause.** `_map_failure`
(`systemd_worker_lifecycle.py:704-717`) collapses `LifecycleConflict`, `StateConflict`,
and `SystemdConflict` into one string,
`("conflict", "operator_recovery", "retained lifecycle facts conflict")`. Separately,
`_register` (`:531-542`) and `_terminate` (`:544-554`) each wrap every
non-`EvidenceRejected`, non-deadline exception as `_AuthorityUnavailable`, which
`_map_failure` renders as
`dependency_unavailable / restore_database / "database authority is unavailable"` — so a
registration refused by the still-active fence (`IncarnationConflict`,
`src/kdive/worker_lifecycle/authority_store.py:42`, raised at `:185` on a unique
violation and reaching `_register` through `PostgresAuthority.register`,
`systemd_worker_lifecycle.py:151-162`) is reported as a database outage while the
database is healthy. `diagnostics`, the designated escape hatch, returned an empty
payload on the wedged host: `_WITHHELD_TEMPLATE`
(`systemd_diagnostics.py:31`) names only the slot, and the two withhold paths that
append `""` instead (`:394-396` unsafe text, `:401-403` a redaction-forbidden value in
the report) leave no text at all. No path names why. `operator_recovery` appears nowhere
in `docs/` or `deploy/`.

Found on a provisioned Ubuntu 26.04.1 KVM host while running the live tiers. Present on
`main`.

## Goals

- A slot whose unit was restarted out of band is reconcilable through the shipped contract,
with no hand-editing of root-owned files and no manual `UPDATE` against the database.
- An out-of-band restart produces a distinct, greppable disposition rather than a generic
invariant failure or a misattributed database-outage message.
- Each conflict class and each authority failure keeps its discriminating cause in the
response, so the reported cause points at the real one.
- `operator_recovery` is defined in operator documentation with the procedure that clears a
wedged slot.
- The fencing decision — what the gate will and will not accept after a restart — is
recorded in an ADR that amends or supersedes ADR-0574.

## Non-goals

- Resident supervision / slot auto-restart — ADR-0574
- Kubernetes lifecycle-witness and Docker/Compose worker authority — those deployment
surfaces / ADR-0536
- `worker_incarnations` schema or fence-protocol change — ADR-0536 / protocol-3 capture
cutover
- Diagnostics redaction and byte-budget rework — ADR-0574 diagnostics bounds
- Stopping `needrestart`/unattended-upgrades sweeping the units —
`deploy/ansible/roles/live_vm_host`
- MCP tool, job-queue, provider-plane changes — not implicated

Two boundary notes on the list above. Entry 2's re-derive branch would let a restarted slot
return without operator action; that is reconciliation of an existing retained generation,
not the resident supervision or slot auto-restart the first non-goal excludes — nothing
gains a `Restart=` policy or a watchdog. And entry 5 touches diagnostics *text*, not the
redaction rules or the byte budgets: R5 is bounded to a fixed-form reason, accounted inside
the emission bounds already documented at
`docs/operating/runbooks/live-stack.md:298-300`.

## Requirements

- **R1.** A same-boot invocation/generation mismatch for the retained unit is treated as
terminal evidence for the retained invocation, so `stop` and `start` reconcile the slot
instead of failing with `LifecycleConflict`. The surrounding rules are unchanged: a
differing boot ID still yields `killed` (`:661-662`), a `BootObservation` on the retained
boot still yields `SystemdUnavailable` (`:663-664`), and an unreadable boot ID still
yields no rule at all.
- **R2.** The gate's behavior after a restart is decided and implemented: either the marker
is re-derived or re-published for the current `INVOCATION_ID`, or the binding stays
strict and the gate emits a distinct out-of-band-restart disposition that an operator and
a log filter can tell apart from a tampering failure. Entry 2's own acceptance is the
emitted disposition, not end-to-end slot recovery — under the strict branch, recovery
arrives with entry 4.
- **R3.** `_map_failure` maps `LifecycleConflict`, `StateConflict`, and `SystemdConflict` to
distinct messages, and `_register` and `_terminate` stop mapping authority refusals —
including `IncarnationConflict` — onto `dependency_unavailable / restore_database`. A
genuine database outage keeps reporting `dependency_unavailable / restore_database`
across `status` and `stop`, as `tests/live_vm/test_systemd_worker_lifecycle.py:318-321`
asserts. Each split message gets a unit test pinning it: `rg` finds no test in `tests/`
holding either current string, so they are unguarded today.
- **R4.** A recovery operation (`recover`, or `stop --force`) exists end to end — contract
`Operation` literal, control socket, coordinator, slot state, shell client — and retires a
slot whose unit is dead by clearing the on-disk slot facts and the `worker_incarnations`
fence in one operation. It covers the residual cases entry 1 cannot reach (see entry 4).
- **R5.** `deploy/systemd/README.md` and `docs/operating/runbooks/live-stack.md` define
`operator_recovery` and give the recovery procedure. `diagnostics` names why a slot was
withheld, using a closed fixed-form reason vocabulary — never an exception string or
captured text — with its bytes accounted inside the existing per-slot and aggregate
emission bounds.
- **R6.** A new ADR under `docs/adr/` records the fencing decision and amends or supersedes
ADR-0574. It states explicitly what a relaxed rule can and cannot release, and — because
entry 4 adds a destructive operation that ADR-0574:130 addresses directly — what a
`recover` operation may clear, what evidence it may not fabricate, and who may invoke it.
`just ci` runs `adr-status-check` (`justfile:472-473,672`), which fails a `Proposed` ADR
cited from `src/` or `tests/`, so the record merges as Accepted in the same PR as the code
that cites it, and ADR-0574's own status bullet is edited there.

## Hazards

These are the reasons this work needs an ADR rather than a patch.

- **Relaxing the marker↔`INVOCATION_ID` binding weakens an accepted fencing invariant at a
trust boundary.** ADR-0574:52-58 states the gate releases a worker only when the marker
proves *this* invocation was registered. A looser rule can release a worker for a
generation that was never registered.
- **Making an invocation mismatch count as termination evidence contradicts ADR-0574:65-66**
("absence within the same host boot is never termination evidence"). A wrong rule
publishes `killed` evidence for a live worker and strands its fence.
- **A `recover` / `stop --force` operation adds a destructive, state-clearing operation to a
privileged root-owned control socket.** ADR-0574:130 calls force removal "an operator
recovery that may strand fences and cannot create termination evidence."
- **Contract skew is enforced, and the enforcement is what breaks first.** The `Operation`
literal (`systemd_worker_contract.py:21`) is a field type of `LifecycleRequest` (`:217`),
and `lifecycle_protocol_identity` (`:323-329`) hashes `LIFECYCLE_PROTOCOL_VERSION` (`:22`)
together with both models' JSON schemas. Adding `recover` therefore changes the protocol
identity, and `require_compatible_lifecycle`
(`scripts/live-stack/worker-lifecycle.sh:30-49`) compares the checkout against the
installed worker venv and hard-fails with `installed lifecycle protocol does not match
this checkout; reprovision the runner` before any request is sent. The argparse choices
(`systemd_worker_control.py:432`), the dispatch `case`
(`worker-lifecycle.sh:305-334`), and `usage()` (`:17-19`) must move with it, and every
provisioned host's worker venv is reinstalled in the same change — the gate and the venv
are shipped by `deploy/systemd/install-live-worker-lifecycle.sh` and the `live_vm_host`
Ansible role, not by a checkout.
- **`ResponseCode` is schema-bearing; the message is not.** `client_exit_status`
(`systemd_worker_contract.py:332-342`, called from `systemd_worker_control.py:424` and
`worker-lifecycle.sh:281,300`) branches on `response.ok` and `response.code` only — it
never reads `response.message`, so splitting the flattened *message* changes no exit
status. `scripts/live-stack/filter-worker-journal-evidence.py:30-45` allowlists worker
journal `msg` regexes and reads neither taxonomy. What is constrained is the code set:
it feeds the protocol identity above, and
`tests/live_vm/test_systemd_worker_lifecycle.py:320-321` asserts specific codes. Split
messages freely; add a `ResponseCode` value only with the skew handling in the previous
bullet.
- **Proof needs a real systemd host.** Unit tests cannot exercise a `daemon-reexec` or an
out-of-band restart; the reconciliation arms have to run against an installed contract on
a provisioned host.

## Sequencing

Entries 1 and 2 must land together or in that order. `install-live-worker-lifecycle.sh` and
the `live_vm_host` role ship the gate, so the gate change and the coordinator change are
coupled and a mixed pair is a skew hazard. The new ADR (carried by entry 1) must amend or
supersede ADR-0574 before the gate's behavior changes under it.

One conditional edge is not in the graph: if entry 1's ADR takes the strict branch for R2,
the epic's first success criterion is not satisfiable until entry 4 merges. Entry 2 is
accepted on its emitted disposition either way; whoever closes the epic checks that
criterion after entry 4, not after entry 2.

## Success criteria

- On a provisioned systemd host, `systemctl restart kdive-live-worker@N.service` followed by
the shipped lifecycle operations returns the slot to a startable state with no manual file
or database edit.
- A wedged slot's `worker_incarnations` row reaches `state=terminated` with
`terminated_at` set through a contract operation, not an `UPDATE`.
- A fence-refused registration reports a fence conflict, not
`"database authority is unavailable"`, while a genuine outage still reports
`dependency_unavailable / restore_database`.
- `diagnostics` on a wedged slot returns a fixed-form reason for the withholding.
- `operator_recovery` resolves to a documented procedure in `deploy/systemd/README.md` and
`docs/operating/runbooks/live-stack.md`.
- `just ci` green; the reconciliation arms proven on a real systemd host with the installed
contract.

## Decomposition

1. **#2485 — Reconcile a same-boot invocation mismatch instead of raising `LifecycleConflict`**
— `_terminal_observation` (`src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py:654-671`)
is the single place that decides whether a retained slot has terminal evidence; the
`LifecycleConflict` raise at `:665` sits three lines after the boot-ID `"killed"` return
at `:662`, with the `BootObservation` → `SystemdUnavailable` branch at `:663-664`
between them, so the rule change is local to that function and the boot-ID, absent-boot,
and membership rules stay exactly as written. Carries the ADR amendment to ADR-0574,
because the rule it changes is the one ADR-0574:65-66 states, and the ADR also has to
answer R6's question about entry 4. Tests:
`tests/processes/lifecycle/systemd/test_systemd_worker_lifecycle.py`. New record under
`docs/adr/`, merged Accepted, with ADR-0574's status bullet edited in the same PR.
*Blocks entries 2, 3.*

2. **#2486 — Decide and implement the worker gate's behavior after an out-of-band restart**
— `_wait_for_release` (`deploy/systemd/bin/kdive-live-worker-gate:118-136`) holds the
whole marker comparison in one function, and the gate is a standalone root-installed
script with no application imports, so its change is separable from the coordinator's.
It is *installed* rather than imported, which is why it moves with, or just after,
entry 1 rather than independently: `deploy/systemd/install-live-worker-lifecycle.sh`,
`deploy/systemd/system/kdive-live-worker@.service`, and
`deploy/ansible/roles/live_vm_host/` ship it. Tests:
`tests/deploy/test_live_worker_gate.py`, which pins the gate's invariant text.
*Blocked by entry 1.*

3. **#2487 — Preserve the discriminating cause across lifecycle conflict and authority failures**
— the flattening is three functions in one module: `_map_failure` (`:704-717`) collapses
three exception classes into
`("conflict", "operator_recovery", "retained lifecycle facts conflict")`, and `_register`
(`:531-542`) and `_terminate` (`:544-554`) are byte-identical `except Exception` wrappers
that re-raise as `_AuthorityUnavailable`. `_terminate` is the path entry 4's recovery
uses to retire the fence, so leaving it out would keep a refused termination reporting a
database outage. All three are response shaping over one taxonomy
(`client_exit_status`, `systemd_worker_contract.py:332`), so they move as one change.
Sequenced after entry 1 because entry 1 changes which conflict classes reach
`_map_failure` at all. Tests:
`tests/processes/lifecycle/systemd/test_systemd_worker_lifecycle.py` (new assertions
pinning each split message) and `tests/live_vm/test_systemd_worker_lifecycle.py:318-321`
(the genuine-outage arm must stay green). Doc: the dependency-failure recovery text at
`docs/operating/runbooks/live-stack.md:302-303`.
*Blocked by entry 1.*

4. **#2488 — Add a recovery operation that retires a dead worker slot and clears its fence**
— the operation surface is one vertical seam with five fixed points that must move
together: the `Operation` literal (`systemd_worker_contract.py:21`) and with it the
protocol identity (`:323-329`), the coordinator in `systemd_worker_lifecycle.py`, the
state clearing in `systemd_worker_state.py`, the argparse choices
(`systemd_worker_control.py:432`), and the dispatch `case` plus `usage()`
(`scripts/live-stack/worker-lifecycle.sh:305-334`, `:17-19`). It is the only entry that
adds a destructive operation to the privileged socket, which is why it is its own issue
rather than a rider on entry 1. **Its residual case is what entry 1 cannot reach.** After
entry 1, an ordinary wedged slot already reconciles through `stop`:
`_post_evidence_cleanup` (`:523-529`) unlinks the slot files via `cleanup_terminated`
(`systemd_worker_state.py:235-256`), and `terminate_worker_incarnation`
(`authority_store.py:242-260`) matches on the retained binding, which for that slot is
the binding the row was registered with. Entry 4 exists for the slots where that path
does not run — an absent or unparseable `state.json`, a rejected evidence write
(`EvidenceRejected`), an unreadable boot ID, a retained binding that no longer matches
the row — and the issue must state that set and prove each case, not re-implement the
entry 1 path. Sequenced after entry 3 so its failures use the split taxonomy. Tests:
`tests/processes/lifecycle/systemd/test_systemd_worker_contract.py`,
`test_systemd_worker_control.py`, `test_systemd_worker_state.py`, and
`tests/scripts/test_live_stack_scripts.py` (which pins the protocol identity and the
client's argument grammar).
*Blocked by entry 3.*

5. **#2489 — Document `operator_recovery` and make `diagnostics` name why it withheld**
— `SystemdWorkerDiagnostics.capture` and `_capture_diagnostics`
(`src/kdive/processes/lifecycle/systemd/systemd_diagnostics.py:300-345`) emit
`_WITHHELD_TEMPLATE` (`:31`), which names the slot and nothing else, and two paths append
`""` instead (`:394-396`, `:401-403`). None of the three says which of the withhold
causes fired. `operator_recovery` has no definition in `deploy/systemd/README.md` or
`docs/operating/runbooks/live-stack.md`. The reason vocabulary stays fixed-form and
closed — one of the causes is precisely that the report held a redaction-forbidden value,
so free text there would re-cross the redaction boundary the non-goals exclude.
Sequenced last because the procedure it documents is the one entry 4 creates.
*Blocked by entry 4.*

## Open questions

- **Entry 2's disposition is a decision, not just an implementation.** Re-deriving the
marker for the current `INVOCATION_ID` restores unattended reconciliation but weakens the
ADR-0574:52-58 binding; keeping it strict preserves the binding but leaves every
out-of-band restart needing entry 4's operation. Entry 1's ADR settles which, and entry 2
implements what the ADR says.
- **Whether the recovery operation is `recover` or `stop --force`** affects the shell
client's argument grammar (`worker-lifecycle.sh` accepts no flags on `stop` today) and the
installed-venv skew window. Entry 4 decides.
- **Whether a relaxed rule needs a second discriminator** — for example requiring that the
retained unit's current invocation be *newer* than the retained one — to keep a replayed
marker from releasing an unregistered generation. Entry 1's ADR should state the answer
either way.

Start with ADR-0574, the lifecycle contract and coordinator in systemd_worker_contract.py and systemd_worker_lifecycle.py, the gate, shell client, control socket, and the cited live VM tests. Trace protocol identity and provisioning through install-live-worker-lifecycle.sh and the live_vm_host role before choosing the reconciliation and recovery design. Done means the contract, state, diagnostics, documentation, ADR, and provisioned-host restart path agree, with no manual state or database edits.

Written by the indexing model from the issue text.