randomparity / randomparity/kdive

feat: fold the bundle sha256 into the external-boot scan pass

Open
#2,570 0 comments 0 reactions 0 assignees View on GitHub
area:build-install effort:M priority:P2 risk:daytime-only status:ready type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem

`_external_boot_evidence` reads the kernel bundle from the object store three
times. The third read exists only to produce `bundle_sha256`: after
`_scan_external_boot_archive` has already streamed the whole compressed object
through `_RangedReader`, `_digest_object` opens a fresh sequential loop over the
same raw bytes and hashes them.

For a 2 GiB bundle that third pass is ~512 additional `get_range` requests and
one more full sequential read, producing a value that could have been
accumulated for free while the scan pass was already fetching those exact bytes.

This is the middle piece of the #2495 split: real work, but no security boundary
moves.

## Evidence

- Third pass call site: `src/kdive/build_artifacts/validation.py:445` —
`"bundle_sha256": _digest_object(store, keys["kernel"], bundle_head.size_bytes)`
inside the evidence dict built by `_external_boot_evidence`
(`src/kdive/build_artifacts/validation.py:408`).
- `_digest_object` hashes the object over its full recorded `size_bytes`:
`src/kdive/build_artifacts/validation.py:463-474`.
- `_RangedReader`, the reader the scan pass already drives over the same object:
`src/kdive/build_artifacts/validation.py:477-543`.
- Scan pass that already consumes the object:
`src/kdive/build_artifacts/validation.py:547`, opening the stream at
`src/kdive/build_artifacts/validation.py:563`
(`tarfile.open(fileobj=..., mode="r|gz")`).
- `bundle_sha256` is a persisted field of the `external-boot-evidence-v1`
document returned at `src/kdive/build_artifacts/validation.py:443-445`.

## Expected

`bundle_sha256` is produced as a side-effect of the existing scan pass, and the
standalone `_digest_object` call for the kernel bundle is removed. For any given
object the resulting digest is byte-identical to what `_digest_object` produces
today, because the value is persisted in `external-boot-evidence-v1` documents
and re-validation of an existing build must not see it change.

The ~512 store requests and the full sequential read of the third pass are gone.

## Proposed approach

Wrap `_RangedReader` in a hashing reader that accumulates `hashlib.sha256()` over
the compressed bytes it serves, and take the digest from that wrapper instead of
calling `_digest_object` at line 445.

Two correctness constraints have to be handled explicitly, and each needs its own
test:

1. **The scan does not consume the whole object.** `tarfile` in `r|gz` streaming
mode stops at the end-of-archive NUL block and does not read the trailing
padding or any bytes after it. A reader that hashes only what the scan
consumed therefore produces a *different* digest than `_digest_object`'s loop
over the full `size_bytes`
(`src/kdive/build_artifacts/validation.py:463-474`). The remainder of the
object must be drained through the same hashing wrapper after the scan
completes. Required test: build an archive with trailing padding after the
end-of-archive marker, run the new path, and assert the digest equals
`_digest_object`'s value for the same object.

2. **Reads are not guaranteed non-overlapping and forward.**
`_RangedReader.seek()` is implemented
(`src/kdive/build_artifacts/validation.py:529-545`) and invalidates the buffer
when the target falls outside the buffered window, so a naive hash-on-every-
`read()` side effect can double-count or skip bytes if the consumer ever
seeks. Hash at the `store.get_range` seam over a monotonically advancing
offset, or otherwise prove the scan path never seeks — do not assume it.

Also add a test that the full `external-boot-evidence-v1` document is
field-for-field equal before and after the change for a representative fixture.

## Non-goals

Operator-approved 2026-09-16, applying to every child split from #2495:

- Changing the `external-boot-evidence-v1` schema or any field value. That is
owned by a separate ADR on the ADR-0583 line; this work requires
value-identical evidence, which is exactly why the digest-equality test above
is the acceptance condition rather than a nicety.
- Relaxing or re-tuning any security threshold
(`_EXTERNAL_BOOT_EXTENSION_MAX_BYTES`,
`_EXTERNAL_BOOT_ARCHIVE_MAX_MEMBERS`, `_EXTERNAL_BOOT_ARCHIVE_MAX_BYTES`,
`_EXTERNAL_BOOT_MEMBER_MAX_BYTES`). Only the enforcement point may move, never
the values.
- Raising `Semaphore(1)` (`src/kdive/services/runs/complete_build.py:47`). That
is a separate admission-control and capacity issue.
- Re-running the ppc64le proof on POWER hardware as an acceptance gate. Tracked
by `docs/debt/0015-ppc64le-finalization-measurement-unrun.md`.
- Optimizing validation paths outside external boot.
- Eliminating the preflight pass. That is the separate, higher-risk child of
#2495 and must not be folded into this change.

## Measurement record

This change removes one of the three measured passes, so the published proof
record
`docs/design/2026-09-14-external-build-finalization-measurement-2318-proof-record.md`
becomes stale for the digest phase and for the total request count once this
lands.

Parent: #2495. Related: #2314.

CAMPAIGN-OCCURRENCE: b6d43c8c0535-e428eada-4e97-4a9e-b1b8-eace09097d9d source=#2495 sweep=#2495
CAMPAIGN-OCCURRENCE-RATIONALE: split of #2495 approved by the operator during campaign triage because the three changes have independent risk profiles

Contributor guide

Open the contributing guide

Research direction

Start in src/kdive/build_artifacts/validation.py with _external_boot_evidence, _digest_object, _RangedReader, and _scan_external_boot_archive. Trace how the r|gz scan reads and seeks the kernel bundle, then add tests for trailing padding, seek-safe hashing, and field-for-field external-boot-evidence-v1 equality. Done means the standalone kernel digest pass is removed, the full-object digest remains identical, and the extra store read is gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.