randomparity / randomparity/kdive

runs.complete_build amplifies S3 reads and ties validation to the MCP request

Open
#2,314 0 comments 0 reactions 0 assignees View on GitHub
area:build-install effort:L priority:P1 risk:daytime-only status:ready type:bug
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem

External-build finalization performs archive validation within the lifetime of a synchronous MCP call. Its unbuffered range reader turns tarfile's small reads into individual S3 GETs, making normal kernel bundles expensive to finalize. There is no durable completion handle to poll when the request times out.

BLACK_BOX_REVIEW.md N-1/N-5 reports six request timeouts: three with a 2 GB artifact and three with a 103 MB artifact, with the Run still created afterwards. Those observations establish the session failure, not a universal size threshold or a measured attribution to decompression.

## Evidence

Source verified at commit 40e70edcb148f396ba536ab017513b96d305ce2a, also the remote main head at review time.

- src/kdive/services/runs/complete_build.py:146 calls preparation, validation, then publication in sequence. At :218 the request awaits a process-local single-slot semaphore and asyncio.to_thread; this is thread offload, not a durable job.
- src/kdive/build_artifacts/validation.py:472 forwards each reader request directly to store.get_range with no read-ahead buffer.
- src/kdive/build_artifacts/validation.py:498 performs a full gzip/tar preflight, then :501 opens another stream with tarfile.open(mode="r|gz") and default buffering. Module hashing at :682 reads decompressed data in up-to-4-MiB chunks; that does not make the underlying S3 requests 4 MiB.
- src/kdive/build_artifacts/validation.py:433 separately hashes the compressed bundle through another object pass.
- src/kdive/store/objectstore.py:708 implements each get_range as one get_object HTTP request.
- src/kdive/mcp/tools/lifecycle/runs/registrar.py:534 tells callers to wait for finalization to answer before re-minting, without exposing a job to observe after a timeout.

A local, in-memory execution of the real validate_external_artifacts with arch="ppc64le", the existing test _boot_elf fixture, and one seeded 1-MiB module produced a valid 1,049,227-byte compressed bundle. Validation succeeded but requested 4,196,910 bytes across 117 range calls, including 102 requests of exactly 10,240 bytes. Every read remained version-pinned. This proves I/O amplification without relying on a slow network or a live kernel. The tar pass alone scales to roughly 10,000 requests for 103 MB before counting other validation passes.

The reports contain no server timings, deployed commit, actual client timeout, or traces distinguishing semaphore wait, I/O, CPU, and database time. The local host is arm64; no live ppc64le timeout reproduction was performed. AGENT_FINDINGS.md's specific timing attribution remains a hypothesis.

## Expected

Valid supported bundles can complete without repeated request-timeout retries, and callers can determine a finalization attempt's outcome after a transport interruption. Validation must retain immutable-version reads, checksums, canonical paths, archive/decompression bounds, module obligations, and atomic publication.

## Proposed approach

1. Fix request amplification at the range-reader seam using bounded buffering. Add a request-count regression with an incompressible fixture and test EOF, short responses, bounds, and version identity.
2. Measure queue wait, store reads, scan time, and publication on representative 103-MB and 2-GB ppc64le bundles against the real MCP client. Do not infer a transport timeout from artifact size.
3. Settle the completion contract from that proof: if supported validation exceeds the request budget, make the same finalization entry point return durable progress that survives disconnect and supports jobs polling. Specify idempotency, cancellation, upload-window fencing, and publication ownership before changing the public or persisted contract.
4. Update the wrapper and served async/upload guidance to match the chosen behavior. Do not remove integrity or archive-safety validation to make the call faster.

Related: closed #2107 introduced the external-boot evidence requirements; closed #1339 addressed a different scan-bound diagnostic. Neither tracks request amplification or timeout recovery.

## Provenance

Filed under the user's request to reconcile BLACK_BOX_REVIEW.md and AGENT_FINDINGS.md from the same ppc64le evaluation against source and create issues for verified problems. Combines N-1 and N-5; N-3's external harness suppression is downstream context, not a KDIVE retry-policy defect.

Contributor guide

Open the contributing guide

Research direction

Start with src/kdive/build_artifacts/validation.py and inspect the range reader around lines 472-501, then trace get_range in src/kdive/store/objectstore.py. Run validate_external_artifacts with the existing _boot_elf fixture and an incompressible bundle, and review complete_build.py and the registrar wrapper. Done requires measured request reduction while preserving validation, followed by an explicit decision and tests for the completion contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
api, backend, cloud, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.