randomparity / randomparity/kdive
External-boot capacity default (256 GiB) exceeds the runner's free space, so reprovisioning fails
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
The external-boot recovery capacity gate demands 256 GiB by default
(`live_vm_host_external_boot_capacity_bytes` 32 GiB × `live_vm_host_external_boot_concurrent_activations` 8).
The self-hosted KVM runner has **194.1 GiB** available and no inventory override, so **re-running
provisioning against it fails the gate today.**
The gate itself is working correctly and its diagnostic is actionable. The defect is that the
shipped defaults do not fit the one host the repository actually provisions with them.
## Observed
On the provisioned runner, using the same measurement the gate makes:
```
$ df -B1 --output=avail /var/lib/kdive
208365330432 # 194.1 GiB
required = 34359738368 * 8 = 274877906944 # 256.0 GiB
```
`194.1 GiB < 256.0 GiB` — the assertion at `local_worker_host`'s
"Require provisioned external-boot recovery capacity before worker release" cannot pass.
Reproduced directly on a second Ubuntu 26.04 host with 221.5 GiB free, which failed the same way,
once per worker slot:
```
TASK [local_worker_host : Require provisioned external-boot recovery capacity before worker release]
failed: (item=kdive-worker-4) => {
"assertion": "item.stdout_lines[-1] | trim | int >= (live_vm_host_external_boot_capacity_bytes | int)
* (live_vm_host_external_boot_concurrent_activations | int)",
"evaluated_to": false,
"msg": "External-boot recovery storage does not meet the required byte capacity. Increase
filesystem space or lower admitted concurrency, then rerun provisioning."
}
```
Passing `-e live_vm_host_external_boot_concurrent_activations=6` (192 GiB) lets provisioning
proceed.
## Evidence
`deploy/ansible/roles/local_worker_host/defaults/main.yml:56-57`:
```yaml
# Per activation; the capacity gate reserves this many bytes for each admitted activation.
live_vm_host_external_boot_capacity_bytes: 34359738368 # 32 GiB
live_vm_host_external_boot_concurrent_activations: 8
```
Neither value is overridden anywhere under `deploy/ansible/inventory/` — not in
`group_vars/live_vm_runners.yml`, not in the runner's `host_vars`. So the runner's effective
requirement is the full 256 GiB.
The 8 here is also not the same 8 as the worker-slot count. `live_vm_host_worker_accounts` lists
eight accounts with a comment insisting the slot count stay literal, but this variable is an
*admitted concurrency* bound for recovery storage. They are currently equal, which makes it easy
to assume lowering one lowers the other.
## Why it matters
Same consequence as #2561, reached by a different route: the host that runs the native `live_vm`
tier cannot be reprovisioned from the repository as it stands. Two independent gates now block a
rebuild — a missing package declaration and this capacity default — and neither is visible until
someone attempts one.
It also means the gate is not currently doing the job it was written for. A check that the only
real host cannot pass is one that will be routinely overridden on the command line, which is how
capacity guards stop being believed.
## Expected
A default the intended hardware satisfies, and the runner's actual bound recorded where it can be
read. Any of:
1. Set `live_vm_host_external_boot_concurrent_activations` in `group_vars/live_vm_runners.yml` to a
value the runner's disk supports, so the fleet's real bound is declared rather than passed by
hand.
2. Lower the shipped default, if 8 concurrent activations was aspirational rather than measured.
3. Grow the runner's filesystem, if 8 × 32 GiB is the genuine requirement — in which case the
provisioning docs should state the disk floor, because nothing does today.
Whichever is chosen, the relationship between this bound and the eight worker slots is worth one
sentence in the defaults, since they are equal by coincidence rather than by construction.
## Out of scope
- The missing `python3-dev` declaration — #2561. Same reproducibility theme, unrelated fix.
- Whether 32 GiB per activation is the right reservation. This issue is about the product of the
two values against real disk, not about either value's derivation.
## Provenance
Found while provisioning a clean host to run a live proof for an unrelated pull request. The
runner's own free space and the absence of an inventory override were then checked directly, so
the claim that a rebuild would fail is measured rather than extrapolated from the second host.
Contributor guide
Research direction
Start with deploy/ansible/roles/local_worker_host/defaults/main.yml:56-57 and inspect deploy/ansible/inventory/group_vars/live_vm_runners.yml plus the local_worker_host capacity assertion. Compare the effective defaults with the measured /var/lib/kdive space, then verify provisioning succeeds without a command-line override and that the relationship to the eight worker slots is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ansible, yaml
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100