randomparity / randomparity/kdive
Decide and implement the worker gate's behavior after an out-of-band restart
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
`deploy/systemd/bin/kdive-live-worker-gate` binds the root-owned release marker to systemd's
`INVOCATION_ID` for the gate's own process. A restart of `kdive-live-worker@N.service` mints
a new `INVOCATION_ID` while the on-disk marker still holds the previous one, so the gate
exits 1 before the worker ever runs, with a message that reads as a tampering failure:
```
kdive-live-worker-gate: kdive worker gate slot N: release marker binding invariant failed
```
Nothing rewrites the marker on a restart, so the slot cannot come back on its own. Two
dispositions are possible and the choice is the ADR's (see the blocking issue); this issue
implements what the ADR decides.
## Evidence
`_wait_for_release` (`deploy/systemd/bin/kdive-live-worker-gate:118-136`) holds the whole
comparison in one function:
- `:120-127` — read the marker, poll until the deadline if absent.
- `:133` — `expected = f"{generation}\n{invocation}\n".encode("ascii")`.
- `:134-135` — any inequality calls `_fail(slot, "release marker binding invariant failed")`.
- `_fail` at `:50-52` raises `SystemExit(1)`.
`invocation` comes from the gate's own environment, so it is guaranteed to differ after a
restart. The generation half of the marker is not what fails here; the invocation half is.
The gate is a standalone root-installed script that loads no application code, so its change
is separable from the coordinator's. What it is *not* separable from is installation: it is
shipped, not imported, so an old installed gate can face a new coordinator.
## Scope
- `deploy/systemd/bin/kdive-live-worker-gate` — implement the ADR's disposition.
- `deploy/systemd/system/kdive-live-worker@.service`
- `deploy/systemd/install-live-worker-lifecycle.sh`
- `deploy/ansible/roles/live_vm_host/`
- `tests/deploy/test_live_worker_gate.py` — this file pins the gate's invariant text, so any
new message lands here.
## The two dispositions
**Re-derive / re-publish** the marker for the current `INVOCATION_ID`. A restarted slot then
returns without operator action. Cost: it weakens the fencing invariant ADR-0574:52-58 states
— the gate releases a worker only when the marker proves *this* invocation was registered —
and a looser rule can release a worker for a generation that was never registered.
**Stay strict** and emit a distinct, greppable out-of-band-restart disposition instead of the
generic invariant failure, so an operator and a log filter can tell it apart from tampering.
Cost: every out-of-band restart then needs the recovery operation added by a later issue in
this epic.
This is reconciliation of an existing retained generation either way. Neither branch adds a
`Restart=` policy or a watchdog; resident supervision and slot auto-restart remain out of
scope (ADR-0574).
## Acceptance
- The gate's behavior after an out-of-band restart matches what the ADR decided, with the
ADR cited in the change.
- Under the strict branch, the out-of-band-restart case emits a message distinct from the
tampering case, and `tests/deploy/test_live_worker_gate.py` pins both.
- Under the re-derive branch, the gate still refuses a marker whose *generation* half does
not match, and still refuses an absent marker past the deadline.
- The installer and the `live_vm_host` role ship the changed gate; a host reprovisioned from
the role gets it.
- Proven on a provisioned systemd host: `systemctl restart kdive-live-worker@N.service` and
observe the gate's disposition. Unit tests cannot exercise a real restart.
- `just ci` green.
## Sequencing
Blocked by the rule-and-ADR issue: the ADR decides which disposition this implements, and
gate and coordinator are coupled through installation, so they land together or gate-after-
coordinator.
This issue's acceptance is the emitted disposition, not end-to-end slot recovery. If the ADR
takes the strict branch, a wedged slot does not become recoverable until the epic's recovery
operation merges — that criterion belongs to the epic, checked after that issue rather than
after this one.
Blocked by #2485
Contributor guide
Research direction
Read blocking issue #2485 and ADR-0574 first to determine the selected disposition. Then inspect deploy/systemd/bin/kdive-live-worker-gate, its service and installer files, the live_vm_host role, and tests/deploy/test_live_worker_gate.py. Run the focused gate tests; done means the ADR behavior, installation paths, messages, host reprovisioning, and systemd-host restart evidence satisfy the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100