randomparity / randomparity/kdive
Project venv sync is gated on checkout change, so a venv broken by a failed run never self-heals
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
The project venv's dependency sync is gated on the git checkout having advanced:
```yaml
# deploy/ansible/roles/live_vm_host/tasks/main.yml
- name: Build the venv against the SYSTEM interpreter (ABI-match python3-guestfs)
ansible.builtin.command:
cmd: "{{ live_vm_host_uv_bin }} sync --python /usr/bin/python3 --group live"
chdir: "{{ live_vm_venv }}"
when: live_vm_host_checkout is changed
```
That condition is a proxy for "the venv might be out of date". It cannot see a venv left broken by
a **failed** earlier run, because a failed run still leaves the checkout at the requested revision.
Re-running provisioning is therefore not a repair: the git task reports unchanged, the sync is
skipped, and the broken venv survives every subsequent run.
## Observed
Reproduced end to end on a freshly-imaged host:
1. **First run** — git checkout advances (`changed`), `uv sync` runs and **fails** building
`libvirt-python` (a missing host header, filed as #2561). The venv is left containing only
virtualenv scaffolding.
2. **Second run**, after fixing the host dependency — the git checkout is already at the requested
revision, so it reports `unchanged`, `uv sync` is **skipped**, and the venv is never repaired.
3. Provisioning then fails later, in its own verification step:
```
TASK [live_vm_host : Run check-local-libvirt.sh as the service account]
FAIL worker venv (/opt/kdive/.venv/bin/python) cannot 'import guestfs, drgn'
(local-libvirt kdump capture, ADR-0203)
=== local-libvirt host is NOT ready (see FAIL entries above) ===
```
The venv at that point contains five entries — `_virtualenv.pth`, `_virtualenv.py`, `__pycache__`,
and the two libguestfs files a *later, unconditional* task symlinks in:
```
$ ls /lib/python3.14/site-packages/
__pycache__ _virtualenv.pth _virtualenv.py guestfs.py libguestfsmod.cpython-314-x86_64-linux-gnu.so
```
`import guestfs` succeeds — that task is unconditional. `import drgn` raises
`ModuleNotFoundError`, because the only task that would install it is the skipped one. The two
failure modes sit side by side in the same venv, which is a fairly direct demonstration of what the
condition costs.
## Why the existing rationale does not cover this
The task's own comment explains the condition, and its reasoning is sound as far as it goes:
> Run whenever the checkout advanced (main is a moving ref), NOT `creates:`-gated — a creates
> guard would freeze the venv's deps at first-provision while the source moved on, so a live
> dependency added to main would never reach the venv the worker imports from. **On a converged
> host (main unchanged) the git task is unchanged, so uv sync is skipped and the run is
> 0-changed.**
It reasons about two states — checkout advanced, and converged host — and a partially-provisioned
host is neither. The comment argues correctly against `creates:` and then lands on a condition with
the same blind spot in a different place.
## Relationship to #2532 / PR #2542
PR #2542 removes exactly this kind of condition from the **lifecycle witness** venv
(`/opt/kdive-live-worker-lifecycle/.venv`), and its test says why in terms that describe this issue
precisely:
> Unconditional: its `when:` proxies could not see a venv left stale by a failed earlier run, and
> the protocol assertion in verify.yml now makes that state unrecoverable.
That change does not touch the project venv task above, which keeps
`when: live_vm_host_checkout is changed`. So after #2542 lands, one of the two venvs the role
builds self-heals and the other does not.
This issue is the sibling gap, filed separately rather than folded into #2542 because that PR is
already at hand-off with a published attestation, and widening it would invalidate that.
## Expected
The project venv converges to the declared dependency set on every run, rather than only when the
checkout moved. Options:
1. **Make the sync unconditional**, matching what #2542 does for the witness venv. `uv sync` is
already a no-op when the venv matches the lock, so the "0-changed on a converged host" property
is preserved by `changed_when` rather than by skipping the command.
2. **Condition on the venv's observed state** instead of on a proxy — e.g. verify the venv imports
what the contract requires, and sync when it does not.
(1) is the smaller change and is consistent with the direction #2542 already takes.
## Out of scope
- The witness venv — #2542.
- The missing `python3-dev` that caused the first run to fail — #2561. Any first-run failure
reaches this state; that one merely happened to be the trigger.
## Provenance
Found while provisioning a clean host to run a live proof for #2542 itself. Not constructed: the
first run failed for an unrelated reason, and re-running provisioning failed to repair it, which is
what surfaced the condition. Confirmed by reading the task, the skipped-task status in the run
output, and the venv contents on the host.
Contributor guide
Research direction
Start in deploy/ansible/roles/live_vm_host/tasks/main.yml at the project venv task, then compare its behavior with the lifecycle venv change described in PR #2542. Re-run provisioning with the checkout already at the requested revision and confirm that a broken venv is repaired and the later verification can import guestfs and drgn.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100