randomparity / randomparity/kdive
feat: raise _RANGE_CHUNK_BYTES to cut external-boot validation store requests
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem
`_RANGE_CHUNK_BYTES` is 4 MiB. Every pass of external-boot validation reads the
kernel bundle through `_RangedReader`, which issues one object-store `get_range`
per chunk. For a 2 GiB bundle that is ~512 requests per pass, and #2495 measured
~1484 requests across the three passes of a single finalization.
Raising the constant to 8 or 16 MiB halves or quarters the request count of every
pass at once, without changing what any pass reads or computes. At the 13.53
ms/request observed on the ppc64le loopback store this is ~10 s saved per
finalization at 8 MiB; on a network-attached store with higher per-request
latency the saving grows with the latency.
This is the smallest and most independent of the three changes split out of
#2495. It has no evidence-value change and no security tradeoff: the chunk size
governs only how the same bytes are fetched, not which bytes are read or which
limits are enforced.
## Evidence
- Constant under change: `src/kdive/build_artifacts/validation.py:57`
(`_RANGE_CHUNK_BYTES = 4 * 1024 * 1024`).
- `_RangedReader` fetches exactly one `store.get_range` per buffer fill and caps
the buffered window at `_RANGE_CHUNK_BYTES`:
`src/kdive/build_artifacts/validation.py:477-543` — see the `fetch_length`
computation and the `if fetch_length <= _RANGE_CHUNK_BYTES` buffering branch.
- `_digest_object` uses the same constant for its own sequential loop:
`src/kdive/build_artifacts/validation.py:463-474`.
- `_discard_exact` in the preflight also reads in `_RANGE_CHUNK_BYTES` steps:
`src/kdive/build_artifacts/validation.py:692-698`.
- Concurrency ceiling that bounds the buffer footprint:
`src/kdive/services/runs/complete_build.py:47`
(`_EXTERNAL_BUILD_VALIDATION_SLOTS = asyncio.Semaphore(1)`).
- The read-ahead buffer this constant sizes was introduced by #2317 (closed);
this issue re-tunes its value, it does not reintroduce the mechanism.
## Expected
`_RANGE_CHUNK_BYTES` is raised to 8 MiB or 16 MiB, halving or quartering the
object-store request count of every validation pass. Validation results —
`bundle_sha256`, `vmlinuz_sha256`, member counts, byte totals, and every
`external-boot-evidence-v1` field — are byte-identical to the current
implementation for the same object.
The constant carries a comment stating the resident buffer footprint the new
value implies and why it is acceptable under `Semaphore(1)`: at most one
validation runs at a time, so the worst-case resident buffer is one chunk per
live `_RangedReader` rather than one per concurrent finalization. That comment is
the record a future change to the semaphore has to read before raising
concurrency.
## Proposed approach
1. Raise `_RANGE_CHUNK_BYTES` at `validation.py:57` to the chosen value.
2. Add the buffer-footprint comment tying the value to the `Semaphore(1)`
admission limit at `complete_build.py:47`.
3. Confirm the existing request-count and reader-boundary regressions in
`tests/providers/local_libvirt/test_validate_external_artifacts.py` still pass
and are not asserting the old constant's value; adjust only a test that
hard-codes 4 MiB as an expectation of the chunk size itself, never one that
asserts a validation outcome.
> **Blocked as filed (2026-09-16).** This issue states that the chunk size "governs only how the same bytes are fetched, not which bytes are read". That is false while #2476's second defect stands: `_magic_offsets` shares its stream with the loop body, so the constant changes which bundles validate. Reproduced on a fixed input - decoy gzip magic at 1000, real gzip-ELF at 6000 - where 1024/2048/4096 reject and 8192/16384/100000 accept. Acceptance criterion 2 (byte-identical validation results) is unreachable until #2476 lands. Evidence is in https://github.com/randomparity/kdive/issues/2476#issuecomment-5703098803.
>
> The 8 MiB value chosen during the attempt stands on its own reasoning and is recorded in the trajectory: request reduction is sublinear while footprint is linear, and 16 MiB collides numerically with `_BANNER_SCAN_CHUNK_BYTES`, `_MAX_SECTION_BYTES` and `_EXTERNAL_BOOT_ELF_METADATA_MAX_BYTES`.
>
> Two corrections to the attempt's own design spec, both verified, so they are not lost: the claim that the `get_range` count strictly falls as the constant rises is false once the chunk reaches the object size, since every loop clamps to `min(chunk, remaining)` (measured on a 1000-byte object: 250->4, 500->2, 1000->1, 2000->1); and the real resident peak is about five chunk-sized buffers live at once, so ~40 MiB at 8 MiB rather than the ~24 MiB the spec states - which matters because that number is written into the source comment a future semaphore raise would read.
Blocked by #2476
## Non-goals
Operator-approved 2026-09-16, applying to every child split from #2495:
- Changing the `external-boot-evidence-v1` schema or any field value. That is
owned by a separate ADR on the ADR-0583 line; this work requires
value-identical evidence.
- Relaxing or re-tuning any security threshold
(`_EXTERNAL_BOOT_EXTENSION_MAX_BYTES`,
`_EXTERNAL_BOOT_ARCHIVE_MAX_MEMBERS`, `_EXTERNAL_BOOT_ARCHIVE_MAX_BYTES`,
`_EXTERNAL_BOOT_MEMBER_MAX_BYTES`). Only the enforcement point may move, never
the values — and this child moves neither.
- Raising `Semaphore(1)`. That is a separate admission-control and capacity
issue.
- Re-running the ppc64le proof on POWER hardware as an acceptance gate. Tracked
by `docs/debt/0015-ppc64le-finalization-measurement-unrun.md`.
- Optimizing validation paths outside external boot.
## Measurement record
This change re-measures `store_wait_ms` for every phase that reads through
`_RangedReader`, so the published proof record
`docs/design/2026-09-14-external-build-finalization-measurement-2318-proof-record.md`
becomes stale for the store-read phase once this lands. The record's request
counts (1484 store requests for a 2 GiB bundle) no longer describe the shipped
code.
Parent: #2495. Related: #2314, #2317.
CAMPAIGN-OCCURRENCE: b6d43c8c0535-e428eada-4e97-4a9e-b1b8-eace09097d9d source=#2495 sweep=#2495
CAMPAIGN-OCCURRENCE-RATIONALE: split of #2495 approved by the operator during campaign triage because the three changes have independent risk profiles
Contributor guide
Research direction
Start with blocked issue #2476 and its linked reproduction before changing the chunk size. Read src/kdive/build_artifacts/validation.py around _RANGE_CHUNK_BYTES, _RangedReader, _digest_object, and _discard_exact, then run the validation tests in tests/providers/local_libvirt/test_validate_external_artifacts.py. Done means the dependency is resolved, request-count and boundary regressions pass, and validation results remain byte-identical.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100