Buck TS action staging exceeds namespace runner disk/time budget as admissions grow (blocks #1209)

Buck TS action staging exceeds namespace runner disk/time budget as admissions grow (blocks #1209)

Part of #1147. Companion: decision 0025 (CoW/reflink local disk economics) and 03-materialization/.delta/DELTA-001-assembler-hardlinks-pending-0025.md.

Motivation

Phase 3 widening is gated by CI lanes that can no longer run the admitted graph. PR #1209 (notion top authority: 4 packages, 16 edge deletions, root solutions 27→23) is red on buck2 + typecheck while its content is proven good by every check that can run:

• stock tsc -p on all four packages: 5–8 s, clean (md 8.3 s, react 5.5 s, ds-sync 7.5 s, cli 4.3 s)
• genie:check + genie:buck2:test green locally
• one retry of the failed lanes reproduced the identical failure deterministically — not flake, not luck

Each admission adds roughly ~1 GB retained output + ~1 min staging work per lane, while the Buck lanes on namespace runners execute everything cold (BUCK2_NO_REMOTE_CACHE=1, off-tailnet by policy — see genie/ci-workflow/shared.ts:57 and the "CI/off-tailnet" record in 05-composition/.experiments/2026-08-30-production-cp-a-member-e2e.md), on small disks, under ~30–35 min lane timeouts. Without intervention every further admission makes this worse, and BUCK-R07 (budgets) / BUCK-R08 (disk anti-duplication) become unreachable. This issue exists so the capacity fix lands before the next admission, not after five more red cycles.

RCA

The bottleneck is action staging, not compilation. Per admitted package, Buck runs:

1. package_tree (packages/@overeng/buck2-tools/src/package-tree.ts): cloneTree full-copies node_modules (copyFileSync(..., COPYFILE_FICLONE) — a full data copy wherever reflinks are unsupported, e.g. ext4/overlay), then assertContainedSymlinks walks the entire tree with one realpathSync per symlink.
2. tsgo_emit / tsgo_typecheck (packages/@overeng/buck2-tools/src/typescript-runner.ts, runEmit/runTypecheck): cp of the whole tree into a staging dir, then makeTreeReadOnly — a sequential, single-threaded lstat+chmod per file — then tsgo, then the same crawl in reverse at cleanup.
3. buck-out retains every package_tree output, so retained weight grows ~linearly with the admitted count.

Measured on dev3 (fast NVMe, lightly loaded) with a storybook-closure package tree (927 MB / ~40k entries):

Step	Time
stock tsc -p (compile control)	8 s
full emit runner end-to-end (typescript-runner.ts + pinned tsgo)	41 s wall (≈6 s compile, ≈35 s staging: cp + chmod crawl + cleanup)
package_tree symlink-audit walk alone	35 s

CI evidence (all on the same action family, both lanes):

• run 33806232240 → ENOSPC: no space left on device staging a node_modules tree (typecheck job), with Commands: 279 (cached: 0, remote: 0, local: 279).
• run 33809913585 → notion-md:dist (tsgo_emit) stuck 12+ min, lane canceled at ~33 min (typecheck, buck2).
• rerun of both failed lanes → identical stall on the identical action (typecheck, buck2).

Excluded with evidence: compile pathology (8 s control), md-specific content (tui-react carries the same storybook closure and builds fine — md:dist was simply the action running when the runner fell over), source/config drift (genie:check + unit tests green).

Adjacent hazard found during repro (not the CI bug, recording it here): pnpm's node_modules contains symlink cycles, so any dereferencing copy explodes — a cp -rL trial hit 71 GB in minutes before being killed. Both runners correctly use dereference: false; this is a loaded gun next to the code, not the fired one.

Local repro (dev3-class machine)

# 1. compile control — expect ~8 s, clean
node node_modules/.pnpm/typescript@6.0.3/node_modules/typescript/bin/tsc \
  -p packages/@overeng/notion-md/tsconfig.json --pretty false

# 2. package-tree audit walk — expect ~35 s on fast disk (scales with disk pressure)
time bun packages/@overeng/buck2-tools/src/package-tree.ts \
  --output /tmp/pt-repro/out --node-modules node_modules \
  --file package.json package.json

# 3. full emit action — expect ~40 s wall, ~6 s of it compile:
#    assemble <tree>/ with package files + node_modules + .workspace-siblings,
#    then:
time bun packages/@overeng/buck2-tools/src/typescript-runner.ts emit \
  /nix/store/2dnkwcaq8cysibwlfvh2zjwabl67wsa2-effect-tsgo/bin/tsgo \
  <tree> tsconfig.json dist src/mod.d.ts <out>

(Use hardlinked input assembly, cp -al, so input prep itself stays instant. Clean up /tmp/pt-repro, /tmp/emit-repro afterwards — the experiment is disk-hungry by nature.)

Options and tradeoffs

Baseline: ~35 s staging overhead per action on fast local disk; ~1 GB retained per tree; 279 cold-local actions in the current graph; namespace lanes pay a large pressure multiplier and die at ~30–35 min / small disk.

Option	Mechanism	Expected effect	Cost / risk
A. Hardlink/reflink staging (lands 0025 + DELTA-001)	share tree bytes via CAS/hardlinks instead of per-action full copies	copy + chmod crawl → ~0; retained disk collapses to unique bytes; helps every future admission and dev machines	medium: isolation semantics need design (chmod mutates shared inodes — read-only bind mounts / overlayfs / CoW layers); touches buck2-tools runners + materialization.bzl
B. Self-hosted Buck lanes	run buck2/typecheck where disk + warm cache live, per BUCK-R06/R07 end-state (cf. #1056 milestone: 283/283 cached, 30.6 s fresh-context)	unblocks the whole roadmap immediately; lane times collapse with cache reads	runner topology + tailnet; dev3 root pressure (#2293 — 81% full and climbing under concurrent agent load at time of writing) must be handled first or ENOSPC just moves address
C. Raise lane timeouts / bigger namespace runners	buy headroom	exactly one round of relief; does not change the per-admission curve	recurring cost; re-litigated every few admissions
D. Retain less (buck-out GC between phases, fewer materialized intermediates)	cap peak disk	saves disk, not time; partial relief only	risks invalidating debugging/caching assumptions; needs measurement per lane

Recommendation: B unblocks now, A fixes the curve; C only as a stopgap. Whichever lands, #1209 (open, content-proven, red purely on capacity) is the validation vehicle: it goes green on the chosen path with no content changes.

Acceptance

• #1209 green without content changes on the chosen path
• per-admission staging budget recorded (p95 action staging time, peak lane buck-out) and added to the R07/R16 benchmark record
• decision record or DELTA entry for the chosen option (amend 0025 or file the successor)

Start with packages/@overeng/buck2-tools/src/package-tree.ts and packages/@overeng/buck2-tools/src/typescript-runner.ts, focusing on cloneTree, runEmit, runTypecheck, and makeTreeReadOnly. Run the documented local repro to measure staging time and disk use, then review decision 0025 and DELTA-001 before choosing an option. Done means #1209 is green without content changes, staging budgets are recorded in the R07/R16 benchmark record, and the decision or DELTA entry is updated.