randomparity / randomparity/kdive

Native live_vm job never installs, so the self-hosted runner drifts until a fail-closed guard catches it

Open Beginner friendly
#2,548 0 comments 0 reactions 0 assignees View on GitHub
area:build-install effort:S priority:P1 type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

The scheduled native `live_vm` job checks out the repository and runs the proofs against whatever build the self-hosted runner was last provisioned with. It never installs. The `live_vm_tcg` job does, on every run.

`.github/workflows/live.yml`, native job — four steps:

1. Checkout
2. Run both native families (reaper → stack → mint → preflight → test, one shell)
3. Capture worker lifecycle diagnostics
4. Clean up live stack

The tcg job carries an install step; the native job has no equivalent. So the runner's installed tree drifts from `main` indefinitely, and nothing in the job notices or corrects it.

## Observed

The runner's source checkout was pinned at a commit from twelve days earlier. The installed worker-lifecycle protocol identity read `1:623185b0…` while the checkout computed `1:c82a1e3b…` — same `LIFECYCLE_PROTOCOL_VERSION`, different schema hash, because ten fields had since been added to `LifecycleRequest`.

`require_compatible_lifecycle` is fail-closed, so every `worker-lifecycle.sh` operation except `diagnostics` (exempt at `worker-lifecycle.sh:182`) refused. It surfaced as the job's `Clean up live stack` step failing with a protocol message, which points at the protocol rather than at the drift that caused it.

Reprovisioning the host fixed it, and the next scheduled run's cleanup step went green.

## Why this recurs, and soon

`lifecycle_protocol_identity()` hashes `LIFECYCLE_PROTOCOL_VERSION` together with `LifecycleRequest.model_json_schema()` and `LifecycleResponse.model_json_schema()`. Any change to those schemas moves the identity even when the version constant does not change.

`LifecycleRequest.operation` is typed `Operation` (`systemd_worker_contract.py:217`), which is a `Literal`. So **adding one operation name moves the identity**. #2532 does exactly that:

```
main: Operation = Literal["start", "status", "stop", "diagnostics"]
#2532: Operation = Literal["start", "status", "stop", "diagnostics", "recover"]
```

with `LIFECYCLE_PROTOCOL_VERSION = 1` on both sides. When that merges, the runner goes stale again the moment it is not reprovisioned by hand.

This is not a rare event. It is any change to the lifecycle request or response shape.

## The asymmetry is the defect

A host that reinstalls every run cannot drift. A host that never reinstalls drifts silently until a fail-closed guard catches it, and then reports the guard rather than the cause. The native tier gets the weaker treatment precisely where the consequences are least visible — a scheduled job nobody is watching at the time.

## Suggested fix

Give the native job an install step, as the tcg job has, so the proof runs against the tree it just checked out.

If reinstalling on every native run is too expensive, the alternative is a cheap guard rather than nothing: compare the installed revision and the computed lifecycle identity against the checkout before the proofs run, and fail with a message naming the drift and the remedy. That converts a confusing mid-run protocol refusal into an actionable preflight failure, consistent with the tri-state policy in `docs/operating/runbooks/live-testing.md:127-136`.

A guard is strictly weaker than installing — it reports drift instead of preventing it — so it is the fallback, not the preference.

## Provenance

Found while diagnosing scheduled native `live_vm` failures. Two other faults on the same host were fixed separately (#2545, #2544); this issue is the reason the second of them existed at all, and the reason it will return. Verified against the contract source rather than inferred: the `Operation` literal comparison above is a direct diff of `main` against #2532's branch.

Contributor guide

Open the contributing guide

Research direction

Open `.github/workflows/live.yml` and compare the native `live_vm` steps with the `live_vm_tcg` install step; start by tracing where checkout and installation occur. Ensure the native job installs the checked-out tree before running both proof families, then verify that the scheduled job's cleanup remains successful without installed/checkout drift.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.