randomparity / randomparity/kdive
Native live_vm job never installs, so the self-hosted runner drifts until a fail-closed guard catches it
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
The scheduled native `live_vm` job checks out the repository and runs the proofs against whatever build the self-hosted runner was last provisioned with. It never installs. The `live_vm_tcg` job does, on every run.
`.github/workflows/live.yml`, native job — four steps:
1. Checkout
2. Run both native families (reaper → stack → mint → preflight → test, one shell)
3. Capture worker lifecycle diagnostics
4. Clean up live stack
The tcg job carries an install step; the native job has no equivalent. So the runner's installed tree drifts from `main` indefinitely, and nothing in the job notices or corrects it.
## Observed
The runner's source checkout was pinned at a commit from twelve days earlier. The installed worker-lifecycle protocol identity read `1:623185b0…` while the checkout computed `1:c82a1e3b…` — same `LIFECYCLE_PROTOCOL_VERSION`, different schema hash, because ten fields had since been added to `LifecycleRequest`.
`require_compatible_lifecycle` is fail-closed, so every `worker-lifecycle.sh` operation except `diagnostics` (exempt at `worker-lifecycle.sh:182`) refused. It surfaced as the job's `Clean up live stack` step failing with a protocol message, which points at the protocol rather than at the drift that caused it.
Reprovisioning the host fixed it, and the next scheduled run's cleanup step went green.
## Why this recurs, and soon
`lifecycle_protocol_identity()` hashes `LIFECYCLE_PROTOCOL_VERSION` together with `LifecycleRequest.model_json_schema()` and `LifecycleResponse.model_json_schema()`. Any change to those schemas moves the identity even when the version constant does not change.
`LifecycleRequest.operation` is typed `Operation` (`systemd_worker_contract.py:217`), which is a `Literal`. So **adding one operation name moves the identity**. #2532 does exactly that:
```
main: Operation = Literal["start", "status", "stop", "diagnostics"]
#2532: Operation = Literal["start", "status", "stop", "diagnostics", "recover"]
```
with `LIFECYCLE_PROTOCOL_VERSION = 1` on both sides. When that merges, the runner goes stale again the moment it is not reprovisioned by hand.
This is not a rare event. It is any change to the lifecycle request or response shape.
## The asymmetry is the defect
A host that reinstalls every run cannot drift. A host that never reinstalls drifts silently until a fail-closed guard catches it, and then reports the guard rather than the cause. The native tier gets the weaker treatment precisely where the consequences are least visible — a scheduled job nobody is watching at the time.
## Suggested fix
Give the native job an install step, as the tcg job has, so the proof runs against the tree it just checked out.
If reinstalling on every native run is too expensive, the alternative is a cheap guard rather than nothing: compare the installed revision and the computed lifecycle identity against the checkout before the proofs run, and fail with a message naming the drift and the remedy. That converts a confusing mid-run protocol refusal into an actionable preflight failure, consistent with the tri-state policy in `docs/operating/runbooks/live-testing.md:127-136`.
A guard is strictly weaker than installing — it reports drift instead of preventing it — so it is the fallback, not the preference.
## Provenance
Found while diagnosing scheduled native `live_vm` failures. Two other faults on the same host were fixed separately (#2545, #2544); this issue is the reason the second of them existed at all, and the reason it will return. Verified against the contract source rather than inferred: the `Operation` literal comparison above is a direct diff of `main` against #2532's branch.
Contributor guide
Research direction
Open `.github/workflows/live.yml` and compare the native `live_vm` steps with the `live_vm_tcg` install step; start by tracing where checkout and installation occur. Ensure the native job installs the checked-out tree before running both proof families, then verify that the scheduled job's cleanup remains successful without installed/checkout drift.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100