stacklok / stacklok/brood-box

Compose rootfs from cached layers at boot instead of storing flattened copies

Open
#85 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
71
Forks
10
Avg merge
4d 8h
Merged PRs (30d)
8

Description

Problem

Every distinct image digest gets its own fully-flattened rootfs directory in the cache (~/.cache/broodbox/images/sha256-*/). The claude-code, codex, and opencode images all share the same wolfi-base layer (~500MB+), but each stores a complete copy. With 3 agent images cached, the shared base content is duplicated 3x on disk.

Additionally, when an image is rebuilt with only a small top-layer change (e.g., updating Claude Code), the entire flattened rootfs is re-extracted and stored as a new entry — even though 95%+ of the content is identical layers already in the layer cache.

Proposal

Instead of flattening all layers into a single rootfs directory at pull time, compose the rootfs at boot time from individually-cached layers. go-microvm already has a LayerCache that stores extracted layers by DiffID — the building blocks are there.

Approach options

Option A — Overlayfs composition (Linux only):
Mount the cached layers as overlayfs lowerdir stack with a tmpfs upperdir. The VM gets a composed view without any copying. Fastest boot, lowest disk usage, but requires overlayfs support and appropriate privileges.

Option B — Reflink-based composition:
At boot time, compose layers bottom-to-top into rootfs-work/ using reflinks (FICLONE). Similar to today's CloneDir of the flattened rootfs, but starting from individual cached layers. Disk usage is reduced because the layer cache is shared, even though the composed result is a full directory tree.

Option C — Lazy composition with hardlinks:
Hardlink files from cached layers into the composed rootfs. Faster than copying, same disk blocks. Caveat: rootfs hooks that modify files in-place would modify the cached layer too — need COW semantics or hook-awareness.

Expected savings
Scenario Current With layer dedup
3 agent images (shared base) ~2.4 GB ~1.0 GB (base stored once)
Image rebuild (top layer only) +800 MB new entry +50 MB new top layer
Where this lives

This is primarily a go-microvm change:

  • image.Cache or a new rootfs.Composer would handle layer-to-rootfs composition
  • microvm.Run() would compose from layers instead of cloning the flattened cache entry
  • The flattened rootfs cache entries (sha256-*) could become optional or be replaced entirely by the layer-only cache

Notes

  • The layer cache (~/.cache/broodbox/images/layers/) already works and is populated during layered extraction
  • Layer ordering and whiteout handling during composition already exists in applyLayerToDir() in go-microvm's image/pull.go
  • This pairs well with layer-aware GC: only layers referenced by at least one cached image config need to be retained

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading go-microvm's image/pull.go, especially applyLayerToDir(), and inspect the existing LayerCache and the microvm.Run() path. Compare the proposed composition options against current flattened-cache behavior, preserving layer ordering and whiteout handling. Done means boot can use cached layers without requiring a separate flattened copy for each image.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
infrastructure, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.