P4: decompose resume (28.58 ms) and destroy (21.31 ms) into their sub-costs

Summary

E10's rung 2 measured the warm hot path as acquire 0.00 ms · resume 28.58 ms · run 2.85 ms · destroy 21.31 ms · total 55.32 ms (authoritative metal run, srv-r16b14s16, 2026-09-15). Resume + destroy are 49.9 of the 55.3 ms and are the entire reason the design's §7.2 stopping rule reads STOP at the 15 ms bar. Right now each is a single black-box number. This issue is about instrumenting both operations internally to find out what inside each is actually expensive, so engineering effort (and any future spec) can target the right sub-step instead of guessing.

Why this matters

• This is the single highest-leverage item in the epic (#256): whatever the eventual latency bar (15, 60, or something else), reducing resume/destroy is the only lever that helps and doesn't require giving up per-Exec isolation (see the epic's "explicitly out of scope" note on parked-shell reuse).
• It likely has a second effect on throughput: replenishment (async standby rebuild) performs a similar restore-shaped operation at a similar order of cost (10.98 ms CPU / ~23 ms wall per restore, from E10 rung 3's "replenish" mode). If resume is what dominates a restore's wall-clock cost, cheaper resume could also push out the c=8 replenishment knee measured in E11 — worth confirming rather than assuming.

What's already known / where to look

• Relevant code: remote-worker/internal/vmpool/launcher_firecracker.go (the VM lifecycle methods — see also the file's comments on SnapshotDir restore and the hardlinked memfile sharing trick), remote-worker/internal/vmpool/fcapi.go (the Firecracker API calls — LoadSnapshot, pause/resume, the mem_backend config), remote-worker/internal/vmpool/pin_linux.go (PinMemoryFile — the MAP_SHARED+mlock of the golden snapshot's memfile that every VMM then maps MAP_PRIVATE).
• Raw data for the current (undecomposed) numbers: E10 rung 2's parked and freshchild JSON records. Parked vs. fresh-child resume times are nearly identical (28.58 vs 28.56 ms), which already rules out "shell-spawn cost inside the guest" as the dominant factor in resume — the cost is almost certainly in the VM-level restore/teardown machinery itself, not guest-side work.
• Candidate sub-costs to isolate for resume: the Firecracker LoadSnapshot API call latency itself; mount-at-acquire (the workspace ext4 image mount — see SerializesExecsPerRun in launcher_firecracker.go); vsock device attach/handshake; jailer/cgroup process spawn; first-touch page faults against the pinned memfile even though it's mlocked (the guest's own page tables still need populating on first access, separate from host page-cache residency).
• Candidate sub-costs for destroy: SIGKILL + process reap; per-VM cgroup teardown (cgroup.go — note the caution there about rmdir vs rm -rf, and see issue #255 for a related leftover-directory finding from the same run); jail/workspace directory cleanup; munmap/munlock bookkeeping if any of it is per-VM rather than shared.

Open questions for the spec

1. What's the right instrumentation approach — in-process phase timers (cheapest, but adds overhead that could itself skew a 20-30 ms measurement), strace/perf at the syscall level, or a flame-graph profiling pass on a representative run?
2. Should this run against the existing E10 driver (deploy/microvm/e10-lifecycle.sh) with added timing hooks, or does it need a dedicated micro-benchmark that isolates resume/destroy from the rest of the lifecycle?
3. Is any part of the cost fixed/unavoidable (e.g., a Firecracker API round-trip floor) vs. tunable (e.g., mount options, cgroup hierarchy depth)?

Suggested scope for a first cut

Add phase-level timing inside the resume and destroy code paths (or wrap them with a profiling harness) for one representative rung, and report a breakdown table analogous to E10's existing acquire/resume/run/destroy split, but one level deeper.

Part of #256 (P4 epic: next experiments after the metal STOP).

Start with deploy/microvm/e10-lifecycle.sh and the VM lifecycle methods in remote-worker/internal/vmpool/launcher_firecracker.go, then inspect fcapi.go, pin_linux.go, and cgroup.go. Add phase-level timing around resume and destroy for one representative E10 rung. Done means a breakdown table decomposing both operations into sub-costs alongside the existing acquire/resume/run/destroy measurements.