randomparity / randomparity/kdive
Support >5 GiB rootfs uploads for local-libvirt Systems via multipart (Option B of #1503)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 26m
- Merged PRs (30d)
- 311
Description
## Problem / motivation
#1503 (PR #1505, ADR-0436) made the System rootfs upload path **reject** chunked/multipart declarations at `create_system_upload` (Option A — fail-fast instead of accept-then-fail-at-provision). That closed the bug, but it leaves the **5 GiB single-PUT ceiling** as a hard cap for agent-uploaded rootfs images. A realistic custom debug rootfs (full distro + kernel-debuginfo + drgn/crash + symbols) routinely exceeds 5 GiB, so the cap is a real capability gap. This issue tracks Option B: actually **supporting** >5 GiB rootfs via server-side multipart reassembly.
## Why this is a feature (design + ADR), not a quick wire-up
Three load-bearing pieces, only the first of which is trivial:
1. **Reassembly wiring (easy).** `reassemble_chunked` (`src/kdive/artifacts/reassembly.py:34`) already exists and is used by `runs.complete_build` (`src/kdive/services/runs/complete_build.py:245`). Wiring it into `_commit_uploaded_rootfs` (`src/kdive/jobs/handlers/systems.py`) is mechanical, and the chunk-rejection guard from #1505 (`_SYSTEM_UPLOAD.allow_chunks=False`) would be removed/replaced.
2. **Integrity model (the real decision).** The #743 install-fetch verifies a **plain whole-object SHA-256** (`src/kdive/providers/local_libvirt/lifecycle/rootfs/rootfs_upload_fetch.py:82-95`): it requires `head.checksum_sha256` and checks `sha256(body) == head.checksum_sha256`. A multipart-completed object's `head.checksum_sha256` is a **composite** (`-`) or `None`, never a plain SHA-256 of the body — so the current verify cannot pass for a reassembled object. The run/build path sidesteps this (it stores `head.etag` at `complete_build.py:333` and never re-verifies whole-object SHA-256 at consumption). System rootfs **does** re-verify at install time, so B must replace this scheme. **Design decision required.**
3. **Streaming (scalability).** The install-fetch loads the entire object into RAM (`store.get_artifact(key).data` → `tmp.write_bytes(data)`, `rootfs_upload_fetch.py:90,101`). For a >5 GiB rootfs that's >5 GiB resident on the provisioning host. A genuine large-file path needs streaming download-to-disk, not load-into-memory.
## Design questions to resolve (spec + ADR)
- **Integrity approach.** (a) Recompute + persist a whole-object SHA-256 at commit *after* reassembly (keeps the existing `sha256(body)==checksum` verify contract; just changes where the stored hash comes from) — but requires reading the whole object server-side once to hash it; or (b) verify per-chunk (reuse `verify_chunks`) before/during reassembly and drop the whole-object re-verify at install; or (c) verify against the composite/per-part scheme. Trade-offs in cost, streaming-compatibility, and how much the install-fetch changes.
- **Streaming install-fetch.** Stream the object to `dest` in bounded chunks while hashing, instead of buffering the whole body. Interacts with (a)/(b).
- **Cap.** `KDIVE_MAX_UPLOAD_BYTES` (50 GiB) already bounds chunked totals; confirm the System rootfs cap and whether it should differ.
- **Live proof.** A real >5 GiB rootfs end-to-end proof (the #743 live proof used 1.3 GiB, leaving the ceiling untested).
## Acceptance (high level; refine in spec)
- A chunked (multipart) `rootfs` declaration >5 GiB is accepted, reassembled server-side, integrity-verified, and provisions a System end-to-end.
- Install-fetch streams the object to disk (no whole-object buffering) and verifies integrity under the chosen scheme.
- The #1505 chunk-rejection guard is replaced by real support (or scoped to a documented ceiling).
- Live/integration proof with a >5 GiB image.
## Context
Follow-up from #1503 / PR #1505 (ADR-0436, Option A guard) and #743 / PR #1499 (ADR-0434, single-PUT rootfs). Multipart reassembly precedent: ADR-0104 (`reassemble_chunked`). Related: remote-libvirt parity #1433 should adopt the same scheme to stay in lockstep. **Deferred to a design session** (integrity-model decision needs human sign-off) alongside #1502/#1504.
Contributor guide
Research direction
Start with ADR-0436, src/kdive/artifacts/reassembly.py, runs/complete_build.py, and the System upload and fetch files named in the issue. Resolve the integrity and streaming design in an ADR before implementing. Done means multipart rootfs uploads over 5 GiB are reassembled, verified, streamed to disk, and proven through an end-to-end integration or live test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100