randomparity / randomparity/kdive
feat: reduce complete_build validation from three store passes to one
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
`runs.complete_build` reads the kernel bundle from the object store three times in sequence, each time from the beginning. For a 2 GiB bundle on a loopback store this costs ~112 s on POWER9 and ~35 s on x86_64 (measured in PR #2477). On a network-attached object store those times grow proportionally to per-request latency because the pass count multiplies the request count, not just the bytes.
The three passes all live in `_external_boot_evidence()` (`src/kdive/build_artifacts/validation.py:408`):
1. **`_preflight_external_boot_archive()`** (line 642) — a full decompressing scan that reads every tar header and payload byte in order to bound GNU/PAX extension sizes before `tarfile` is handed the stream.
2. **`_scan_external_boot_archive()`** (line 547) — a second full decompressing scan via `tarfile.open(mode="r|gz")` that hashes `boot/vmlinuz`, validates every module member, and builds the module source manifest.
3. **`_digest_object()`** (line 463, called at line 445 for the kernel bundle) — a third sequential read of the raw compressed bytes to produce the bundle `sha256`.
Each pass makes ~512 range GET requests against the object store for a 2 GiB bundle at the current `_RANGE_CHUNK_BYTES = 4 MiB` (line 57), for ~1484 requests total (measured). The ppc64le proof run recorded `scan_ms = 112 153 ms` against `total_ms = 112 201 ms` — the scan is the entire cost.
The per-byte rate on ppc64le (54.52 ns/byte) vs x86_64 (18.87 ns/byte) is a machine-speed factor on gzip decompression and sha256 — not a different attribution. The ppc64le row uses 37.4% of the 300 s supported client budget vs 11.6% for x86_64. The budget still holds, but the margin narrows on slower hardware and disappears entirely under 8+ concurrent finalizations (the semaphore at `src/kdive/services/runs/complete_build.py:47` serializes them).
## Evidence
- Three-pass structure: `src/kdive/build_artifacts/validation.py:408–445` (`_external_boot_evidence` calling all three in sequence)
- Pass 1 (preflight decompress): `src/kdive/build_artifacts/validation.py:642–689`
- Pass 2 (scan decompress): `src/kdive/build_artifacts/validation.py:547–639`
- Pass 3 (bundle sha256): `src/kdive/build_artifacts/validation.py:463–474`
- Chunk size constant: `src/kdive/build_artifacts/validation.py:57` (`_RANGE_CHUNK_BYTES = 4 MiB`)
- `_RangedReader` (one store GET per chunk): `src/kdive/build_artifacts/validation.py:477–521`
- Validation runs in `asyncio.to_thread` under a `Semaphore(1)`: `src/kdive/services/runs/complete_build.py:47,294–309`
- Measurement instrumentation and proof that scan = total cost: `src/kdive/services/runs/complete_build.py:442,451`
- ppc64le proof row (PR #2477, committed 2026-09-15): `docs/design/2026-09-14-external-build-finalization-measurement-2318-proof-record.md` — `scan_ms 112 153` of `total_ms 112 201`, 37.4% of 300 s budget, 1484 store requests
## Expected
Validation of a 2 GiB kernel bundle should require at most one full decompressing pass over the object and one sequential read of the compressed bytes, eliminating ~66% of store requests and one full CPU decompress cycle. The security properties preserved by the current three-pass structure (gzip bomb guard, PAX extension cap, member count limit, bundle sha256, module member hashes) must all be retained.
## Proposed approach
Three independent changes, in increasing scope. Each is independently shippable:
**A. Hash compressed bytes as a side-effect of the scan pass (eliminates pass 3)**
Replace the standalone `_digest_object` call at line 445 with a `_HashingRangedReader` that wraps `_RangedReader` and accumulates a `hashlib.sha256()` on every `get_range()` call. The bundle sha256 is produced as a side-effect of the pass 2 scan, at zero additional store cost. Removes ~512 store requests per finalization.
**B. Merge `_preflight_external_boot_archive` into `_scan_external_boot_archive` (eliminates pass 1)**
The preflight's security assertions — GNU/PAX extension payload cap (`_EXTERNAL_BOOT_EXTENSION_MAX_BYTES`), member count ceiling (`_EXTERNAL_BOOT_ARCHIVE_MAX_MEMBERS`), and raw decompressed-byte total (`_EXTERNAL_BOOT_ARCHIVE_MAX_BYTES`) — can be checked inline during the `tarfile` streaming pass. The PAX concern (a malformed extension causing `tarfile` to allocate before any limit fires) can be addressed by intercepting the extension payload before `tarfile` processes it or by running a bounded header-only scan that does not decompress member payloads. Removes ~512 store requests and one full gzip decompress per finalization.
**C. Larger `_RANGE_CHUNK_BYTES` (reduces request count, orthogonal to A and B)**
Increasing `_RANGE_CHUNK_BYTES` from 4 MiB to 8 or 16 MiB halves or quarters the number of store GETs per pass, reducing `store_wait_ms` proportionally. At 13.53 ms/request on the ppc64le loopback store, halving requests saves ~10 s per finalization. On a network-attached store with higher per-request latency this effect grows significantly. One-line change, no security tradeoff.
A and B together reduce the pass count from 3 to 1, cutting ~1000 of the ~1484 measured store requests and one full decompress cycle. C is orthogonal and compounds with both.
Contributor guide
Research direction
Start in src/kdive/build_artifacts/validation.py at _external_boot_evidence() and read _preflight_external_boot_archive(), _scan_external_boot_archive(), _digest_object(), and _RangedReader. Trace the call from src/kdive/services/runs/complete_build.py and review the documented measurement record. Done means retaining the listed security checks and bundle and module hashes while reducing validation to one decompressing pass plus one sequential compressed-byte read.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100