randomparity / randomparity/kdive

Project venv sync is gated on checkout change, so a venv broken by a failed run never self-heals

Open Beginner friendly
#2,564 0 comments 0 reactions 0 assignees View on GitHub
area:provisioning effort:S priority:P2 risk:night-safe type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem

The project venv's dependency sync is gated on the git checkout having advanced:

```yaml
# deploy/ansible/roles/live_vm_host/tasks/main.yml
- name: Build the venv against the SYSTEM interpreter (ABI-match python3-guestfs)
ansible.builtin.command:
cmd: "{{ live_vm_host_uv_bin }} sync --python /usr/bin/python3 --group live"
chdir: "{{ live_vm_venv }}"
when: live_vm_host_checkout is changed
```

That condition is a proxy for "the venv might be out of date". It cannot see a venv left broken by
a **failed** earlier run, because a failed run still leaves the checkout at the requested revision.
Re-running provisioning is therefore not a repair: the git task reports unchanged, the sync is
skipped, and the broken venv survives every subsequent run.

## Observed

Reproduced end to end on a freshly-imaged host:

1. **First run** — git checkout advances (`changed`), `uv sync` runs and **fails** building
`libvirt-python` (a missing host header, filed as #2561). The venv is left containing only
virtualenv scaffolding.
2. **Second run**, after fixing the host dependency — the git checkout is already at the requested
revision, so it reports `unchanged`, `uv sync` is **skipped**, and the venv is never repaired.
3. Provisioning then fails later, in its own verification step:

```
TASK [live_vm_host : Run check-local-libvirt.sh as the service account]
FAIL worker venv (/opt/kdive/.venv/bin/python) cannot 'import guestfs, drgn'
(local-libvirt kdump capture, ADR-0203)
=== local-libvirt host is NOT ready (see FAIL entries above) ===
```

The venv at that point contains five entries — `_virtualenv.pth`, `_virtualenv.py`, `__pycache__`,
and the two libguestfs files a *later, unconditional* task symlinks in:

```
$ ls /lib/python3.14/site-packages/
__pycache__ _virtualenv.pth _virtualenv.py guestfs.py libguestfsmod.cpython-314-x86_64-linux-gnu.so
```

`import guestfs` succeeds — that task is unconditional. `import drgn` raises
`ModuleNotFoundError`, because the only task that would install it is the skipped one. The two
failure modes sit side by side in the same venv, which is a fairly direct demonstration of what the
condition costs.

## Why the existing rationale does not cover this

The task's own comment explains the condition, and its reasoning is sound as far as it goes:

> Run whenever the checkout advanced (main is a moving ref), NOT `creates:`-gated — a creates
> guard would freeze the venv's deps at first-provision while the source moved on, so a live
> dependency added to main would never reach the venv the worker imports from. **On a converged
> host (main unchanged) the git task is unchanged, so uv sync is skipped and the run is
> 0-changed.**

It reasons about two states — checkout advanced, and converged host — and a partially-provisioned
host is neither. The comment argues correctly against `creates:` and then lands on a condition with
the same blind spot in a different place.

## Relationship to #2532 / PR #2542

PR #2542 removes exactly this kind of condition from the **lifecycle witness** venv
(`/opt/kdive-live-worker-lifecycle/.venv`), and its test says why in terms that describe this issue
precisely:

> Unconditional: its `when:` proxies could not see a venv left stale by a failed earlier run, and
> the protocol assertion in verify.yml now makes that state unrecoverable.

That change does not touch the project venv task above, which keeps
`when: live_vm_host_checkout is changed`. So after #2542 lands, one of the two venvs the role
builds self-heals and the other does not.

This issue is the sibling gap, filed separately rather than folded into #2542 because that PR is
already at hand-off with a published attestation, and widening it would invalidate that.

## Expected

The project venv converges to the declared dependency set on every run, rather than only when the
checkout moved. Options:

1. **Make the sync unconditional**, matching what #2542 does for the witness venv. `uv sync` is
already a no-op when the venv matches the lock, so the "0-changed on a converged host" property
is preserved by `changed_when` rather than by skipping the command.
2. **Condition on the venv's observed state** instead of on a proxy — e.g. verify the venv imports
what the contract requires, and sync when it does not.

(1) is the smaller change and is consistent with the direction #2542 already takes.

## Out of scope

- The witness venv — #2542.
- The missing `python3-dev` that caused the first run to fail — #2561. Any first-run failure
reaches this state; that one merely happened to be the trigger.

## Provenance

Found while provisioning a clean host to run a live proof for #2542 itself. Not constructed: the
first run failed for an unrelated reason, and re-running provisioning failed to repair it, which is
what surfaced the condition. Confirmed by reading the task, the skipped-task status in the run
output, and the venv contents on the host.

Contributor guide

Open the contributing guide

Research direction

Start in deploy/ansible/roles/live_vm_host/tasks/main.yml at the project venv task, then compare its behavior with the lifecycle venv change described in PR #2542. Re-run provisioning with the checkout already at the requested revision and confirm that a broken venv is repaired and the later verification can import guestfs and drgn.

Written by the indexing model from the issue text.

Assessment

Tech stack
ansible, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.