randomparity / randomparity/kdive

Decide and implement the worker gate's behavior after an out-of-band restart

Open
#2,486 0 comments 0 reactions 0 assignees View on GitHub
area:core-platform effort:M priority:P1 risk:daytime-only status:ready type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem

`deploy/systemd/bin/kdive-live-worker-gate` binds the root-owned release marker to systemd's
`INVOCATION_ID` for the gate's own process. A restart of `kdive-live-worker@N.service` mints
a new `INVOCATION_ID` while the on-disk marker still holds the previous one, so the gate
exits 1 before the worker ever runs, with a message that reads as a tampering failure:

```
kdive-live-worker-gate: kdive worker gate slot N: release marker binding invariant failed
```

Nothing rewrites the marker on a restart, so the slot cannot come back on its own. Two
dispositions are possible and the choice is the ADR's (see the blocking issue); this issue
implements what the ADR decides.

## Evidence

`_wait_for_release` (`deploy/systemd/bin/kdive-live-worker-gate:118-136`) holds the whole
comparison in one function:

- `:120-127` — read the marker, poll until the deadline if absent.
- `:133` — `expected = f"{generation}\n{invocation}\n".encode("ascii")`.
- `:134-135` — any inequality calls `_fail(slot, "release marker binding invariant failed")`.
- `_fail` at `:50-52` raises `SystemExit(1)`.

`invocation` comes from the gate's own environment, so it is guaranteed to differ after a
restart. The generation half of the marker is not what fails here; the invocation half is.

The gate is a standalone root-installed script that loads no application code, so its change
is separable from the coordinator's. What it is *not* separable from is installation: it is
shipped, not imported, so an old installed gate can face a new coordinator.

## Scope

- `deploy/systemd/bin/kdive-live-worker-gate` — implement the ADR's disposition.
- `deploy/systemd/system/kdive-live-worker@.service`
- `deploy/systemd/install-live-worker-lifecycle.sh`
- `deploy/ansible/roles/live_vm_host/`
- `tests/deploy/test_live_worker_gate.py` — this file pins the gate's invariant text, so any
new message lands here.

## The two dispositions

**Re-derive / re-publish** the marker for the current `INVOCATION_ID`. A restarted slot then
returns without operator action. Cost: it weakens the fencing invariant ADR-0574:52-58 states
— the gate releases a worker only when the marker proves *this* invocation was registered —
and a looser rule can release a worker for a generation that was never registered.

**Stay strict** and emit a distinct, greppable out-of-band-restart disposition instead of the
generic invariant failure, so an operator and a log filter can tell it apart from tampering.
Cost: every out-of-band restart then needs the recovery operation added by a later issue in
this epic.

This is reconciliation of an existing retained generation either way. Neither branch adds a
`Restart=` policy or a watchdog; resident supervision and slot auto-restart remain out of
scope (ADR-0574).

## Acceptance

- The gate's behavior after an out-of-band restart matches what the ADR decided, with the
ADR cited in the change.
- Under the strict branch, the out-of-band-restart case emits a message distinct from the
tampering case, and `tests/deploy/test_live_worker_gate.py` pins both.
- Under the re-derive branch, the gate still refuses a marker whose *generation* half does
not match, and still refuses an absent marker past the deadline.
- The installer and the `live_vm_host` role ship the changed gate; a host reprovisioned from
the role gets it.
- Proven on a provisioned systemd host: `systemctl restart kdive-live-worker@N.service` and
observe the gate's disposition. Unit tests cannot exercise a real restart.
- `just ci` green.

## Sequencing

Blocked by the rule-and-ADR issue: the ADR decides which disposition this implements, and
gate and coordinator are coupled through installation, so they land together or gate-after-
coordinator.

This issue's acceptance is the emitted disposition, not end-to-end slot recovery. If the ADR
takes the strict branch, a wedged slot does not become recoverable until the epic's recovery
operation merges — that criterion belongs to the epic, checked after that issue rather than
after this one.

Blocked by #2485

Contributor guide

Open the contributing guide

Research direction

Read blocking issue #2485 and ADR-0574 first to determine the selected disposition. Then inspect deploy/systemd/bin/kdive-live-worker-gate, its service and installer files, the live_vm_host role, and tests/deploy/test_live_worker_gate.py. Run the focused gate tests; done means the ADR behavior, installation paths, messages, host reprovisioning, and systemd-host restart evidence satisfy the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.