randomparity / randomparity/kdive
feat: fold the bundle sha256 into the external-boot scan pass
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
`_external_boot_evidence` reads the kernel bundle from the object store three
times. The third read exists only to produce `bundle_sha256`: after
`_scan_external_boot_archive` has already streamed the whole compressed object
through `_RangedReader`, `_digest_object` opens a fresh sequential loop over the
same raw bytes and hashes them.
For a 2 GiB bundle that third pass is ~512 additional `get_range` requests and
one more full sequential read, producing a value that could have been
accumulated for free while the scan pass was already fetching those exact bytes.
This is the middle piece of the #2495 split: real work, but no security boundary
moves.
## Evidence
- Third pass call site: `src/kdive/build_artifacts/validation.py:445` —
`"bundle_sha256": _digest_object(store, keys["kernel"], bundle_head.size_bytes)`
inside the evidence dict built by `_external_boot_evidence`
(`src/kdive/build_artifacts/validation.py:408`).
- `_digest_object` hashes the object over its full recorded `size_bytes`:
`src/kdive/build_artifacts/validation.py:463-474`.
- `_RangedReader`, the reader the scan pass already drives over the same object:
`src/kdive/build_artifacts/validation.py:477-543`.
- Scan pass that already consumes the object:
`src/kdive/build_artifacts/validation.py:547`, opening the stream at
`src/kdive/build_artifacts/validation.py:563`
(`tarfile.open(fileobj=..., mode="r|gz")`).
- `bundle_sha256` is a persisted field of the `external-boot-evidence-v1`
document returned at `src/kdive/build_artifacts/validation.py:443-445`.
## Expected
`bundle_sha256` is produced as a side-effect of the existing scan pass, and the
standalone `_digest_object` call for the kernel bundle is removed. For any given
object the resulting digest is byte-identical to what `_digest_object` produces
today, because the value is persisted in `external-boot-evidence-v1` documents
and re-validation of an existing build must not see it change.
The ~512 store requests and the full sequential read of the third pass are gone.
## Proposed approach
Wrap `_RangedReader` in a hashing reader that accumulates `hashlib.sha256()` over
the compressed bytes it serves, and take the digest from that wrapper instead of
calling `_digest_object` at line 445.
Two correctness constraints have to be handled explicitly, and each needs its own
test:
1. **The scan does not consume the whole object.** `tarfile` in `r|gz` streaming
mode stops at the end-of-archive NUL block and does not read the trailing
padding or any bytes after it. A reader that hashes only what the scan
consumed therefore produces a *different* digest than `_digest_object`'s loop
over the full `size_bytes`
(`src/kdive/build_artifacts/validation.py:463-474`). The remainder of the
object must be drained through the same hashing wrapper after the scan
completes. Required test: build an archive with trailing padding after the
end-of-archive marker, run the new path, and assert the digest equals
`_digest_object`'s value for the same object.
2. **Reads are not guaranteed non-overlapping and forward.**
`_RangedReader.seek()` is implemented
(`src/kdive/build_artifacts/validation.py:529-545`) and invalidates the buffer
when the target falls outside the buffered window, so a naive hash-on-every-
`read()` side effect can double-count or skip bytes if the consumer ever
seeks. Hash at the `store.get_range` seam over a monotonically advancing
offset, or otherwise prove the scan path never seeks — do not assume it.
Also add a test that the full `external-boot-evidence-v1` document is
field-for-field equal before and after the change for a representative fixture.
## Non-goals
Operator-approved 2026-09-16, applying to every child split from #2495:
- Changing the `external-boot-evidence-v1` schema or any field value. That is
owned by a separate ADR on the ADR-0583 line; this work requires
value-identical evidence, which is exactly why the digest-equality test above
is the acceptance condition rather than a nicety.
- Relaxing or re-tuning any security threshold
(`_EXTERNAL_BOOT_EXTENSION_MAX_BYTES`,
`_EXTERNAL_BOOT_ARCHIVE_MAX_MEMBERS`, `_EXTERNAL_BOOT_ARCHIVE_MAX_BYTES`,
`_EXTERNAL_BOOT_MEMBER_MAX_BYTES`). Only the enforcement point may move, never
the values.
- Raising `Semaphore(1)` (`src/kdive/services/runs/complete_build.py:47`). That
is a separate admission-control and capacity issue.
- Re-running the ppc64le proof on POWER hardware as an acceptance gate. Tracked
by `docs/debt/0015-ppc64le-finalization-measurement-unrun.md`.
- Optimizing validation paths outside external boot.
- Eliminating the preflight pass. That is the separate, higher-risk child of
#2495 and must not be folded into this change.
## Measurement record
This change removes one of the three measured passes, so the published proof
record
`docs/design/2026-09-14-external-build-finalization-measurement-2318-proof-record.md`
becomes stale for the digest phase and for the total request count once this
lands.
Parent: #2495. Related: #2314.
CAMPAIGN-OCCURRENCE: b6d43c8c0535-e428eada-4e97-4a9e-b1b8-eace09097d9d source=#2495 sweep=#2495
CAMPAIGN-OCCURRENCE-RATIONALE: split of #2495 approved by the operator during campaign triage because the three changes have independent risk profiles
Contributor guide
Research direction
Start in src/kdive/build_artifacts/validation.py with _external_boot_evidence, _digest_object, _RangedReader, and _scan_external_boot_archive. Trace how the r|gz scan reads and seeks the kernel bundle, then add tests for trailing padding, seek-safe hashing, and field-for-field external-boot-evidence-v1 equality. Done means the standalone kernel digest pass is removed, the full-object digest remains identical, and the extra store read is gone.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100