randomparity / randomparity/kdive

Support >5 GiB rootfs uploads for local-libvirt Systems via multipart (Option B of #1503)

Open
#1,507 2 comments 0 reactions 0 assignees View on GitHub
area:mcp-api area:provisioning effort:M priority:P2 provider:local-libvirt risk:daytime-only status:ready type:feature
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 26m
Merged PRs (30d)
311

Description

## Problem / motivation

#1503 (PR #1505, ADR-0436) made the System rootfs upload path **reject** chunked/multipart declarations at `create_system_upload` (Option A — fail-fast instead of accept-then-fail-at-provision). That closed the bug, but it leaves the **5 GiB single-PUT ceiling** as a hard cap for agent-uploaded rootfs images. A realistic custom debug rootfs (full distro + kernel-debuginfo + drgn/crash + symbols) routinely exceeds 5 GiB, so the cap is a real capability gap. This issue tracks Option B: actually **supporting** >5 GiB rootfs via server-side multipart reassembly.

## Why this is a feature (design + ADR), not a quick wire-up

Three load-bearing pieces, only the first of which is trivial:

1. **Reassembly wiring (easy).** `reassemble_chunked` (`src/kdive/artifacts/reassembly.py:34`) already exists and is used by `runs.complete_build` (`src/kdive/services/runs/complete_build.py:245`). Wiring it into `_commit_uploaded_rootfs` (`src/kdive/jobs/handlers/systems.py`) is mechanical, and the chunk-rejection guard from #1505 (`_SYSTEM_UPLOAD.allow_chunks=False`) would be removed/replaced.

2. **Integrity model (the real decision).** The #743 install-fetch verifies a **plain whole-object SHA-256** (`src/kdive/providers/local_libvirt/lifecycle/rootfs/rootfs_upload_fetch.py:82-95`): it requires `head.checksum_sha256` and checks `sha256(body) == head.checksum_sha256`. A multipart-completed object's `head.checksum_sha256` is a **composite** (`-`) or `None`, never a plain SHA-256 of the body — so the current verify cannot pass for a reassembled object. The run/build path sidesteps this (it stores `head.etag` at `complete_build.py:333` and never re-verifies whole-object SHA-256 at consumption). System rootfs **does** re-verify at install time, so B must replace this scheme. **Design decision required.**

3. **Streaming (scalability).** The install-fetch loads the entire object into RAM (`store.get_artifact(key).data` → `tmp.write_bytes(data)`, `rootfs_upload_fetch.py:90,101`). For a >5 GiB rootfs that's >5 GiB resident on the provisioning host. A genuine large-file path needs streaming download-to-disk, not load-into-memory.

## Design questions to resolve (spec + ADR)

- **Integrity approach.** (a) Recompute + persist a whole-object SHA-256 at commit *after* reassembly (keeps the existing `sha256(body)==checksum` verify contract; just changes where the stored hash comes from) — but requires reading the whole object server-side once to hash it; or (b) verify per-chunk (reuse `verify_chunks`) before/during reassembly and drop the whole-object re-verify at install; or (c) verify against the composite/per-part scheme. Trade-offs in cost, streaming-compatibility, and how much the install-fetch changes.
- **Streaming install-fetch.** Stream the object to `dest` in bounded chunks while hashing, instead of buffering the whole body. Interacts with (a)/(b).
- **Cap.** `KDIVE_MAX_UPLOAD_BYTES` (50 GiB) already bounds chunked totals; confirm the System rootfs cap and whether it should differ.
- **Live proof.** A real >5 GiB rootfs end-to-end proof (the #743 live proof used 1.3 GiB, leaving the ceiling untested).

## Acceptance (high level; refine in spec)

- A chunked (multipart) `rootfs` declaration >5 GiB is accepted, reassembled server-side, integrity-verified, and provisions a System end-to-end.
- Install-fetch streams the object to disk (no whole-object buffering) and verifies integrity under the chosen scheme.
- The #1505 chunk-rejection guard is replaced by real support (or scoped to a documented ceiling).
- Live/integration proof with a >5 GiB image.

## Context

Follow-up from #1503 / PR #1505 (ADR-0436, Option A guard) and #743 / PR #1499 (ADR-0434, single-PUT rootfs). Multipart reassembly precedent: ADR-0104 (`reassemble_chunked`). Related: remote-libvirt parity #1433 should adopt the same scheme to stay in lockstep. **Deferred to a design session** (integrity-model decision needs human sign-off) alongside #1502/#1504.

Contributor guide

Open the contributing guide

Research direction

Start with ADR-0436, src/kdive/artifacts/reassembly.py, runs/complete_build.py, and the System upload and fetch files named in the issue. Resolve the integrity and streaming design in an ADR before implementing. Done means multipart rootfs uploads over 5 GiB are reassembled, verified, streamed to disk, and proven through an end-to-end integration or live test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.