Design: on-disk, layer-deduplicated OCI image cache for atelet

Consolidated design addressing #437, #120, #166, #228, and unblocking #223/#276/#235/#220. Supersedes the flattened-per-digest cache sketched in #228 with a per-layer store (same overlayfs consumption model, adds cross-image layer dedup).

## Problem

Image handling in the atelet has no persistent cache and duplicates work at every level. Current behavior (`cmd/atelet/internal/memorypullcache/memorypullcache.go`, `cmd/atelet/oci.go`):

- Images are pulled with go-containerregistry and immediately **flattened** via `mutate.Extract`, destroying layer identity.
- The only cache is an **in-memory** LRU of whole flattened tarballs: digest-keyed only (**tag refs are never cached**), bounded by entry count with no byte accounting, lost on atelet restart, shared with nothing.
- The rootfs is **fully re-untarred on every actor run/resume** (`prepareOCIDirectory` → `untar`; `resetActorDirs` wipes the bundle dir between runs).
- Layers shared between images are downloaded, stored, and unpacked once **per image**, not once per node.

Measured impact from existing issues:

- Resume latency is ~99% rootfs extraction: untar ~15–20 s vs. `runsc restore` ~268 ms (#166, #228).
- The in-memory cache's 100 MiB guard never fires — `v1.Image.Size()` returns *manifest* size — and the count-bounded LRU retains one flattened copy of every digest ever pulled: atelet RSS grew 0.8 → 15.3 GiB over 4 days, OOMKilled, stranding actors in `STATUS_RESUMING` (#437).
- `mutate.Extract` itself spikes memory on multi-GB images (all layer decompressors/response bodies held open); observed OOM at a 64 GiB pod limit (#120).
- Cold pulls of large images death-loop against the 28 s workflow deadline (#233).

At target scale (up to hundreds of images per node, ~1.5 GB compressed each, ~40% shared layers), flattened per-image storage needs ~1 TB where layer dedup needs ~650 GB — and every shared base layer is re-downloaded per image.

Related TODOs: `memorypullcache.go:51`; roadmap "Shared image cache", "Peer-to-peer OCI and snapshot sharing".

## Proposed design

A content-addressed, on-disk **layer pool** owned solely by the atelet (single-writer — no daemon, no leases DB; precedent: the digest-named asset cache in `cmd/atelet/sandbox_assets.go`).

### On-disk layout

```
/ # flag, NOT /run (see #30); default under /var/lib/atelet
layers/sha256// # unpacked layer trees (overlayfs lowerdirs) — stored ONCE per node
manifests/sha256/.json # manifest + config + ordered diffid list + recorded sizes
refs/ # tag → digest resolution cache (TTL)
pins/ # expiring preload leases
version # layout version marker
```

Layers shared by N images exist once; an "image" is only a manifest record listing its layers in order. Compressed blobs are not retained after unpack (revisit with p2p sharing). Everything survives atelet restart and node reboot. Startup recovery: sweep orphaned `*.tmp-*` dirs, load manifest index, rebuild mount pins from `/proc/mounts`, drop expired preload pins. Running actors are unaffected by atelet restarts (overlay mounts are kernel state) — this also removes the #437 collateral where a restart mid-restore strands actors.

Sizing note for operators: the cache root must live on a volume sized for both capacity *and* IOPS — on GKE, disk size gates IOPS, and an undersized volume throttles unpack throughput (cf. #30's `/run` trap).

### Pull path (replaces `MemoryPullCache.Fetch` + `mutate.Extract`)

1. Resolve ref → manifest digest (`remote.Head` for tags — fixes tag refs bypassing the cache, and provides the resolved-digest return value #223 needs). Record multi-arch index→platform mapping under both digests (fixes the keying caveat at `memorypullcache.go:183`).
2. For each layer missing from the pool: download → verify digest → decompress → untar in one stream into `layers/sha256/.tmp-`, atomic rename. Parallel layer downloads (bounded, ~4). Memory use is O(stream buffers), independent of image size — closing #120's failure mode structurally.
3. Singleflight by diffid and by manifest digest: concurrent actor starts of overlapping images never duplicate a download or unpack.
4. Layer unpack via **`containerd/containerd/archive.Apply`** (correct OCI whiteout → overlayfs conversion, opaque xattrs, path safety) — as suggested in the #120 comments. Do not hand-roll.
5. Auth is an injected authenticator interface, not the current hardcoded GCP special case — leaves the door open for #432/#459 without expanding scope here.

### Rootfs composition (replaces per-run untar)

- Per actor start: read manifest record, assemble `lowerdir=` from layer paths (reversed — OCI lists bottom-first, overlayfs wants top-first), `mkdir` per-actor `rootfs/`, `upper/`, `work/` under the existing bundle path, one `mount` syscall. Milliseconds regardless of image size.
- **gVisor path:** bundle `rootfs/` in `prepareOCIDirectory` becomes the overlay mountpoint; runsc unchanged. Hardlink/reflink alternatives are rejected per #228's analysis (rootfs is writable; reflink is fs-dependent).
- **microVM path:** `ReconstructSharedDirFromImage` mounts a lowerdir-only (implicitly read-only) overlay for virtiofsd; the guest keeps building its own tmpfs upper in `StartOverlayWorkload`. Single-layer images: RO bind mount.
- **Semantics preserved:** per-actor writes land in `upper/`, wiped by `resetActorDirs` between runs — identical "pristine rootfs per run" contract as today. Image content becomes physically immutable (shared RO lowers) while the actor keeps a writable rootfs — the exact "image mount ro + writable overlay" outcome the #235 thread converged on, and the mechanical lower/upper split #220 needs.
- Teardown: unmount `rootfs/` **before** the bundle-dir wipe; decrement layer pins on unmount.
- Known limits (document only): mount-option page-size cap (~60–80 layers with long paths; mitigate via short dir names or `lowerdir+=`), ~500-lowerdir kernel cap.

### Layer materializer interface (future-proofing for image streaming)

Defined from day one: `EnsureLayer(diffid, descriptor) → path`, `Release(diffid)`, `SizeOf(diffid)`; untar backend first. A future lazy-pull backend (eStargz/SOCI-style FUSE serving the layer path, cf. `stargz-store`) plugs in without redesign; full pull remains the fallback for non-streamable images. Invariants: no code outside the materializer walks/mutates/deletes layer dirs or assumes plain directories; all mounts centralized in the atelet.

### GC / eviction

- Watermark-driven (kubelet-style: evict at high watermark, stop at low) plus `--image-cache-max-bytes`. Sizes recorded at unpack time — never `du` at eviction.
- Protection hierarchy, then LRU by image last-use:
1. layers of actively mounted images (in-memory refcounts, rebuilt from `/proc/mounts`) — never evicted;
2. layers of images with an unexpired preload pin — never evicted;
3. everything else — LRU candidates.
- A layer is deleted only when its last retained referencing image is gone.

### Control-plane integration (aligned with the #276 NodeInventory direction)

- **Inventory reporting:** atelet periodically reports cached image digests (and later rootfs/layer state) through the node-bound, TTL-backed **NodeInventory** mechanism proposed in the #276 discussion — soft preference only; stale data costs latency, never correctness. No standalone one-off RPC/store key (per the Phase 0 consensus there; rebase target for PR #298).
- **`PreloadImage(ref, ttl)`** — async; runs the normal pull path with no actor, for when the control plane knows actors will land on a node 1–2 min ahead. Creates an **expiring pin** (default ~10 min, persisted in `pins/`) so GC can't evict before actors arrive; converts to a mount pin on first actor start, lapses to normal LRU on expiry (mispredictions can't leak disk). Actors arriving mid-preload join in-flight downloads via singleflight. Admission check: fail loudly if unique size can't fit even after maximal eviction. On-demand pulls take download-slot priority over preloads. Contract: best-effort; verify via inventory. Also mitigates #233's cold-pull deadline exposure (though the lock-TTL/deadline decoupling in #233 is still needed independently).

## Implementation phases

**Phase 0 — immediate mitigation (independent of the cache):**
- [ ] Bump go-containerregistry to ≥ v0.21.6 (upstream fix for the #120 extract memory spike)

**Phase 1 — core cache (the performance win):**
- [ ] Layer pool + manifest records + tag resolution (`refs/`) + version marker; cache-root flag + sizing docs
- [ ] Streaming pull path: parallel downloads, singleflight, `archive.Apply`, atomic rename; injectable authenticator
- [ ] Materializer interface with untar backend
- [ ] Overlay composition in `prepareOCIDirectory`; unmount-before-wipe ordering vs `resetActorDirs`
- [ ] microVM path: RO overlay into virtiofsd shared dir
- [ ] Startup recovery (tmp sweep, index load, `/proc/mounts` pin rebuild)
- [ ] Delete `memorypullcache` (closes #437's bug class)
- [ ] Spike: validate overlayfs-backed rootfs under `runsc restore` end-to-end (#228 flagged this; expected orthogonal since restore only needs correct rootfs content)

**Phase 2 — GC + observability:**
- [ ] Size accounting, watermark eviction, layer refcounts, preload-pin persistence
- [ ] Metrics/traces: cache hit/miss, bytes downloaded vs reused, rootfs-materialization time (asked for in #228), cache size, evictions

**Phase 3 — control-plane integration:**
- [ ] Report cached digests via NodeInventory (#276 Phase 1/2 shape)
- [ ] `PreloadImage` RPC with expiring pins + admission check + download prioritization

**Later / out of scope:** lazy-pull backend (eStargz/SOCI), blob retention + p2p/GCS sharing (roadmap), generic registry credentials (#432), credential-bundle caching (#459), snapshot-manifest persistence for tag-based ActorTemplates (#223's control-plane half — enabled by this pull path).

## Rejected alternatives

- **containerd daemon:** operational weight, no shim story for the microVM path, against the in-process design.
- **Flattened per-image disk cache** (#228's initial shape, or extending memorypullcache to disk): loses cross-image layer dedup (download + disk + page cache), loses delta-efficient image updates, no per-layer seam for a streaming backend.
- **Hardlink/reflink rootfs cloning:** unsafe (writable rootfs truncates shared inodes) / fs-dependent, per #228's analysis.
- **Persisting per-actor upperdirs:** would silently change the "pristine rootfs per run" actor contract; out of scope.

## Dependencies

Reuse: `go-containerregistry` (already vendored; bump ≥ 0.21.6), `containerd/archive` (new), `image-spec`/`go-digest` (promote from indirect), `x/sync/singleflight`. Nothing transplantable from kubernetes/kubernetes (kubelet delegates image management to the CRI runtime); its watermark GC policy shape is reimplemented trivially.

## Related issues

Closes #437, #166, #228; closes #120 together with the Phase 0 bump. Enables #223, #276 (NodeInventory consumer), #235, #220. Complements #233 (deadline decoupling still required for first-ever cold pulls).