randomparity / randomparity/kdive

Add a recovery operation that retires a dead worker slot and clears its fence

Open
#2,488 0 comments 0 reactions 0 assignees View on GitHub
area:core-platform area:security effort:L priority:P1 risk:daytime-only type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem

Retained worker state spans the filesystem and the database, and no operation reconciles the
two. When a slot's unit is dead and the contract's own reconciliation cannot reach it, the
`worker_incarnations` row stays `state=active, terminated_at=NULL` and holds the
re-registration fence indefinitely. Recovery in #2481 required moving aside root-owned files
under the slot directory **and** running an `UPDATE` against the database by hand.

The shipped interface is `start COUNT|status|stop|diagnostics`
(`scripts/live-stack/worker-lifecycle.sh:17-19`) and none of those clears the state.

## Evidence

**The residual case — what this operation is actually for.** After the epic's
`_terminal_observation` change, an ordinary wedged slot already reconciles through `stop`:
`_post_evidence_cleanup` (`systemd_worker_lifecycle.py:523-529`) unlinks `worker.env`,
`worker-incarnation.credential`, `release`, and `state.json` via `cleanup_terminated`
(`systemd_worker_state.py:235-256`), and `terminate_worker_incarnation`
(`src/kdive/worker_lifecycle/authority_store.py:242-260`) matches on the retained binding,
which for that slot is the binding the row was registered with — so the fence clears too.

This issue exists for the slots that path cannot reach. Name and prove each:

- `state.json` absent, truncated, or unparseable — no retained binding to match on.
- The retained binding no longer matches the row (a binding rewritten out of band).
- `EvidenceRejected` from the authority, so the terminal write never commits.
- An unreadable boot ID, where no termination rule applies at all.

Do not re-implement the reconciliation path the sibling issue already fixed.

**The surface is one vertical seam with five fixed points that must move together:**

- `Operation = Literal["start", "status", "stop", "diagnostics"]`
(`systemd_worker_contract.py:21`), a field type of `LifecycleRequest` (`:217`).
- the coordinator in `systemd_worker_lifecycle.py`.
- the slot-fact clearing in `systemd_worker_state.py`.
- the argparse choices (`systemd_worker_control.py:432`).
- the dispatch `case` (`worker-lifecycle.sh:305-334`) and `usage()` (`:17-19`).

**Contract skew is enforced, and the enforcement breaks first.**
`lifecycle_protocol_identity` (`systemd_worker_contract.py:323-329`) hashes
`LIFECYCLE_PROTOCOL_VERSION` (`:22`) together with `LifecycleRequest.model_json_schema()` and
`LifecycleResponse.model_json_schema()`. Adding an `Operation` value therefore changes the
protocol identity, and `require_compatible_lifecycle`
(`scripts/live-stack/worker-lifecycle.sh:30-49`) compares the checkout against the installed
worker venv and hard-fails with `installed lifecycle protocol does not match this checkout;
reprovision the runner` (`:46`) before any request is sent. The venv and the gate are shipped
by `deploy/systemd/install-live-worker-lifecycle.sh` and the `live_vm_host` Ansible role, not
by a checkout, so every provisioned host is reinstalled as part of this change.
`tests/scripts/test_live_stack_scripts.py` pins the identity and the client's argument
grammar.

**This is a destructive operation on a privileged socket.** ADR-0574:130 calls force removal
"an operator recovery that may strand fences and cannot create termination evidence." The ADR
carried by the epic's first issue is **ADR-0657**
(`docs/adr/0657-a-successor-invocation-is-terminal-evidence.md`), Accepted, whose section at
`:62-66` states what `recover` may clear and what evidence it may not fabricate: it "may clear
the on-disk slot facts and release the `worker_incarnations` fence for a slot proven dead; it may
not fabricate a `TerminationOutcome`, attribute one invocation's exit facts to another, or run
for a slot whose invocation identity is unreadable." Both children cite it; no new ADR is needed.

## Scope

- `src/kdive/processes/lifecycle/systemd/systemd_worker_contract.py`
- `src/kdive/processes/lifecycle/systemd/systemd_worker_control.py`
- `src/kdive/processes/lifecycle/systemd/systemd_worker_lifecycle.py`
- `src/kdive/processes/lifecycle/systemd/systemd_worker_state.py`
- `scripts/live-stack/worker-lifecycle.sh`
- `tests/processes/lifecycle/systemd/test_systemd_worker_contract.py`,
`test_systemd_worker_control.py`, `test_systemd_worker_state.py`
- `tests/scripts/test_live_stack_scripts.py`

## Decision — settled

**`recover` is a new `Operation` value, not `stop --force`.** `worker-lifecycle.sh` accepts no
flags on `stop` today (`:322-328`), so the flag form would change the client's argument grammar
while the new-operation form changes only the `Operation` literal
(`systemd_worker_contract.py:21`). Both change the protocol identity identically, so the grammar
cost decides it.

**The identity move and the fleet reprovision are accepted.** This change moves
`lifecycle_protocol_identity` (`systemd_worker_contract.py:323-329`), so every already-provisioned
host is reprovisioned after it merges. That is a known, accepted consequence, not an open
question. The gate fails closed with the existing skew message
(`worker-lifecycle.sh:46`) rather than half-working, and `diagnostics` is exempt from the guard
(`worker-lifecycle.sh:182`), so a skewed host stays inspectable throughout the window.

**A sixth fixed point, not in the list above.** `_call_lifecycle`
(`systemd_worker_control.py:192-200`) dispatches every non-`start` operation as
`getattr(lifecycle, request.operation)(deadline)`, so the `Lifecycle` protocol (`:88-98`) must
gain a `recover` member alongside the coordinator method or the request is an `AttributeError`.

## Decomposition

The vertical seam cannot be split — a child that adds the `Operation` literal without the
dispatch case, or the argparse choice without the coordinator, is individually unshippable — so
the split is by behaviour layer, with the whole seam and the reinstall in the first child.

1. **#2532 — Add the recover operation across the full lifecycle seam and reinstall every
provisioned host** — all six fixed points, the `recover` semantics for a slot whose retained
`state.json` is readable and whose binding matches, the `reset-failed` clearing ADR-0657:82-83
names, refusing a live unit, publishing no fabricated evidence, the installer and
`live_vm_host` reinstall, and the `tests/scripts/test_live_stack_scripts.py` identity and
grammar update. **This is the only child that moves the protocol identity, so it carries the
reinstall.** `effort:L`. *Blocked by #2487.*

2. **#2533 — Make recover reach the residual slots: absent or malformed state, drifted binding,
rejected evidence** — the four residual cases from the Evidence section above, plus the
explicit refusal for an unreadable invocation identity, which ADR-0657:62-66 forbids
recovering. No `Operation`, request or response change, so no second identity move and no
second reprovision. `effort:M`. *Blocked by #2532.*

## Acceptance

- The operation retires a slot whose unit is dead, clearing the on-disk slot facts and the
`worker_incarnations` fence in one call, for each residual case listed above.
- It refuses a slot whose unit is *live* — it must not clear a fence out from under a running
worker.
- It publishes no fabricated termination evidence; whatever it does to the row is what the
ADR authorizes.
- `LIFECYCLE_PROTOCOL_VERSION` handling and the reinstall are part of this change: a host
reprovisioned from the role gets the matching venv, and a stale installed venv fails
closed with the existing skew message rather than half-working.
- Proven on a provisioned systemd host against the installed contract, starting from a slot
wedged by a real out-of-band restart.
- `just ci` green.

## Sequencing

Blocked by the response-taxonomy issue, so this operation's failures carry the split causes
rather than the flattened `conflict` string. Blocks the documentation issue, which documents
the procedure this creates.

Blocked by #2487

Contributor guide

Open the contributing guide

Research direction

Start with the dependency #2487, then read the lifecycle seam in systemd_worker_contract.py, systemd_worker_control.py, systemd_worker_lifecycle.py, systemd_worker_state.py, and worker-lifecycle.sh. Run the listed lifecycle and script tests, and inspect the installer and live_vm_host role. Done means the recover operation, protocol identity, fleet reinstall, dead-slot safety, and ADR-0657 constraints pass with just ci green.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, bash, python
Domain
devops, infrastructure, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.