RFC: keep writable-layer filestores out of the checkpoint image (O(1) C/R, CoW fork)

### Description

**Background.** We run long-lived agent sandboxes on runsc `--overlay2=all:dir=...` (hours-long lifetimes, GB-scale writable layers) and are building an epoch-level park/unpark scheduler on `runsc checkpoint/restore`: for resource-scheduling reasons, batch workloads are parked (checkpointed and their resources freed), then restored to continue running once resources free up. For that design, checkpoint/restore latency is the scheduling quantum — it has to stay flat as the writable layer grows.

**Problems we measured.**

*1. C/R cost scales linearly with writable-layer size, and the bottleneck is serialization CPU, not I/O.* Today the writable layer (an unlinked sparse filestore kept alive by a gofer fd) is serialized through the sentry's private MemoryFile on every checkpoint: park ~1.05 ms/MB, unpark ~0.54 ms/MB, image size 1:1 with the layer. Placing the image on tmpfs does not speed the checkpoint up *at all* in our measurements — only restore benefits from faster storage. There is no dirty-tracking or incremental interface. Re-verified on bare metal (104 cores, 187G, kernel 6.8, cgroup v2; "upstream" = main built from source, "PR" = our branch, identical workload and md5 verification per cell):

| writable layer | upstream park | upstream restore | upstream image | PR park | PR restore | PR image |
|---|---|---|---|---|---|---|
| 64M | 75–91ms | 226–326ms | 65MB | 87–104ms | 163–196ms | 1MB |
| 256M | 205–227ms | 1.4–2.6s | 257MB | 87–104ms | 163–196ms | 1MB |
| 1G | 709–798ms | 5.7–8.4s | 1025MB | 87–104ms | 163–196ms | 1MB |
| 4G | 2.7–4.0s | 23–28s | 4097MB | 87–104ms | 163–196ms | 1MB |

Upstream is linear in S on both sides of C/R; the PR is flat (the checkpoint pause window itself is 63–68ms, FICLONE included). GB-scale agent layers mean minutes of park latency and image sizes equal to the whole layer, per cycle, on upstream.

*2. Memory cgroup limits make it substantially worse.* With a quota near the layer size, upstream checkpoint degrades another ~3.1x (1236ms unlimited → 4153ms at a 512M quota, 1G layer): the full serialization pass faults every page of the layer through the sandbox's memcg, and sustained reclaim churn dominates. During an 8s checkpoint under a 1G limit, memcg failcnt rises by ~1M; during restore under a real limit, image pages are evicted while being consumed (~2.5x re-reads), and the checkpoint churn itself evicts the freshly written image (post-checkpoint image page residency: 100% unlimited / 95% at 4G / 16% at 1G / 8% at 512M — restore degrades to cold reads). It never OOMs (file pages are reclaimable; oom_kill stays 0), it just gets slow exactly when the sandbox is resource-constrained, which is precisely the overcommitted regime a park/unpark scheduler targets.

Re-verified head-to-head against current upstream main (same bare metal, 2G writable layer, cold page cache before restore, two runs per cell, md5-verified):

| memcg quota | upstream park (max events) | upstream restore (max events) | PR park (max events) | PR restore (max events) |
|---|---|---|---|---|
| unlimited | 0.9–1.3s (0) | 0.9–1.0s (0) | ~65ms (0) | 0.24–0.29s (0) |
| 4G | 1.1–4.0s (≤80) | 1.4–1.5s (≤80) | ~65ms (0) | 0.24–0.29s (0) |
| 1G | 2.6s (3.3–3.8k) | 2.6–2.7s (6.1–6.6k) | ~65ms (≤8) | 0.24–0.29s (0) |
| 512M | 2.6–2.8s (5.7–6.4k) | **8.9–9.6s (8.7–9.8k)** | ~65ms (≤12) | 0.24–0.29s (0) |

(max events = `memory.events:max` delta during the window; the PR park column is the in-checkpoint freeze window — the orchestrator's host `sync` around it is not runsc's cost.) At the 512M quota, upstream cold restore is ~9x the unlimited baseline while the PR is literally indistinguishable across quotas. Lifting the quota for the restore window and re-applying it afterwards recovers upstream's baseline — but in 2 of 4 runs at 512M, the re-applied limit silently killed the sentry after a successful restore (restore exit 0, memory.current collapses to ~8M, sentry process gone, no oom_kill, no panic); the identical sequence passed 4/4 on the PR.

As a side note, the O(S) image has a robustness edge too: near ENOSPC we measured upstream checkpoint silently truncating the pages file (exit 0, 200MB image for a ~1G logical image) with the corruption surfacing later as a restore panic.

**Proposal.** One sentence: the writable layer already *is* a host file — checkpoint should bind its lifecycle to the state image (atomically paired for the park duration) instead of round-tripping its contents through sentry memory. Concretely:

- `checkpoint --skip-filestore-pages`: private (disk-backed) MemoryFiles save segment metadata only; the image becomes a pure memory-state template, opened strictly read-only on restore and therefore reusable;
- the filestore itself survives the park as an external host-side artifact — primarily a reflink (FICLONE) CoW snapshot on XFS/btrfs; for filesystems without reflink (ext4) we prototype an orchestrator-held-fd fallback, which we consider a degraded backup design (single inode, no fork, awkward ownership) rather than the desired interface;
- `restore --filestore-adopt-dir`: adopts the externally kept copy as the writable layer.

**Benefits (all measured end-to-end on this PR, md5-verified).**

1. *O(1) park/restore.* Park 87–104ms, restore 163–196ms (kvm +80ms for vCPU setup) — flat across writable-layer size (64M–4G), quota (0.5x–∞), and platform (systrap/kvm/ptrace). At S=4G that is 34x faster park and 147x faster restore than upstream; at a 512M quota, >30x faster cold restore (8.9–9.6s → 0.24–0.29s).
2. *Natural CoW fork semantics.* The image is a read-only template and each restore adopts a reflink copy, so one checkpoint yields N write-isolated sandboxes: per-instance marginal cost 220–290ms measured flat over both N (1→64) and S (64M–1G), and disk usage grows only with blocks actually written after the clone (0 for read-only instances — N×S "virtual copies" are free). We think this maps directly onto function cold-start, sandbox cloning, and A/B trial runs.
3. *Memory and writable layer managed independently.* The PR's working set lives in the host page cache (file pages), not anon memory inside the sandbox memcg: under 0.5x quotas we measure zero oom_kill *and* zero pgscan — problem 2's churn regime simply does not arise, because there is no O(S) serialization pass faulting the layer through the memcg, and no O(S) image competing for the same memory.
4. *Image shrinks from O(S) to ~1MB.* Memory state only — which also removes the ENOSPC silent-truncation failure mode above.
5. *Cross-node migration decouples.* The two artifacts travel independently: image via object storage (the existing GCS checkpoint gofer precedent), writable layer via reflink on shared storage or delta transfer.

**What our PR changes.** We have a reviewable branch (`scheduler-adopt-filestore`, 8 commits / ~1.8k LoC on top of current master `80336ad54`, open as PR https://github.com/google/gvisor/pull/14228):

- `pgalloc`: `SaveOpts.ExternalContent` (metadata-only save for private MemoryFiles; skips zero-page scanning, which would mutate the backing file via decommit) and `LoadFrom` support for externally-contented MemoryFiles, with loud failures when the backing file was not adopted or is undersized; `MemoryFileOpts.AdoptExistingFile` (do not truncate on create); a guard so destroy no longer punches holes into a filestore saved with external content (measured: upstream teardown zeroes every written page on graceful exit — 16385 → 0 nonzero 4K pages — while SIGKILL leaves it intact; with the guard the artifact survives both paths);
- `runsc checkpoint --skip-filestore-pages` (opt-in) plumbed through control/state save options; `runsc restore --filestore-adopt-dir` adopting host files in mount order; fresh-container starts with adoption enabled are refused loudly (they would truncate the artifacts);
- hardening now landed on the branch: a per-store artifact manifest (`filestores.json`: resource ID, exact size, chunk count, sampled SHA-256 fingerprint) written **inside the checkpoint freeze window** and verified on adopt before the sandbox starts, so a mismatched or swapped filestore fails loudly instead of silently restoring foreign writable-layer data; clone-on-adopt (default on — each restore takes a private FICLONE copy, making one image+snapshot pair safely resumable into N sandboxes); the in-band snapshot (`checkpoint --filestore-snapshot-dir`) taken by the sentry itself inside the pause window, which makes `--skip-filestore-pages` correct under `--leave-running` without relying on orchestrator call-ordering conventions (and avoids the EXDEV / `open_by_handle_at` dance entirely); plus the seccomp rule for exactly that FICLONE ioctl;
- unit tests for the metadata-only roundtrip over the same host file, with loud failures for the non-adopted and undersized cases.

Robustness/performance data behind the PR (bare metal: 104 cores / 187G / kernel 6.8 / cgroup v2, md5-verified on every cell):

- *Platform gate*: systrap / kvm / ptrace all pass C/R + fork (kvm CPU-state probe byte-identical across restore); ext4 without reflink correctly degrades to the fd-hold path and refuses fork.
- *72-cell perf matrix* (3 platforms × writable-layer {64M, 256M, 1G, 4G} × quota {∞, 1x, 0.5x} × {PR, upstream}): PR park 87–104ms (checkpoint ~70ms + FICLONE ~24ms), restore 163–196ms, image ~1MB — flat across all three axes; upstream scales O(S) (at 4G: park 3.0–4.0s, restore 23–28s, image 4097MB). Under 0.5x quotas (enforced memory.max, OOM-kill verified live) both builds show zero oom_kill and zero pgscan — the PR's working set is host page cache, not sandbox anon memory.
- *Concurrency*: 64 sandboxes parked concurrently — zero failures, 64/64 manifests, zero fd/holder leaks, disk delta 0; concurrent restore storm wall-time 686ms.
- *Fork*: single-template fanout to N=64 at 174–206ms per instance (flat in N and S), disk delta 0; 3-generation lineage (grandchild sees all ancestor markers, sibling branches isolated); template md5 unchanged across rounds.
- *Soak & faults*: 100 park→fanout→verify rounds, zero corruption, zero disk drift; fault injection (kill orchestrator at every phase, kill holder, injected checkpoint failure) always converges to "no manifest ⇒ refuse and safe to erase" or "manifest ⇒ snapshot complete and usable" — never a half state; upstream's ENOSPC silent truncation (above) is structurally impossible on the PR (~1MB image).

**Additional observations that shape the interface.**

1. The filestore fd's `f_path.mnt` sits on a detached private bind mount, so `FICLONE`/`linkat` via procfs fail with EXDEV even on the same superblock (Ubuntu 5.15). We work around it with `name_to_handle_at(AT_EMPTY_PATH)` + `open_by_handle_at` (requires CAP_DAC_READ_SEARCH) — any host-side snapshot orchestrator will hit this; the in-band snapshot above avoids it entirely.
2. A sandbox with multiple writable mounts has multiple anonymous filestores with nothing host-side mapping an fd to its mount (random `runsc-filestore-*` names); orchestrators currently guess via fd ordering, which is exactly why we are adding the signed artifact manifest.
3. Cross-node restore hits two implicit gates even with identical builds: the statefile embeds the source host's CPUID FeatureSet (`CheckHostCompatible`, strict, no flag — the sentry being pure userspace means features like vmx can never execute, yet old→new fails on vmx/hle/rtm and new→old on ~46 avx512/amx features), and the build version string must match exactly. Heterogeneous kernels were fine (5.15 ↔ 6.8 both ways).

**Questions for the community.**

- Is externalizing the writable layer's C/R (lifecycle binding instead of content serialization) a direction upstream would take? What are the main objections — consistency, lifecycle ownership, k8s compatibility? Is dirty-tracking/incremental checkpointing on the roadmap as an alternative?
- For the interface: would signed artifact manifests + adopt-dir be acceptable as the contract, or does upstream prefer a managed/named filestore mode (or an authoritative fd↔mount mapping) for external orchestrators?
- Is constant-cost multi-instance restore from a read-only template + CoW snapshots (fork) worth first-class support? On our side it is the main semantic win beyond O(1) park/restore.
- Is the teardown punch-hole behavior considered contractual? If external liveness is legitimate, should destroy skip decommit for externally-saved filestores (as our guard does)?
- Any appetite for a park-time fleet-baseline CPU mask (à la QEMU live-migration) and a looser version-compatibility policy for cross-node restore?

Start by reviewing PR #14228 and the mentioned pgalloc SaveOpts.ExternalContent and MemoryFileOpts.AdoptExistingFile changes, then inspect the runsc checkpoint and restore flag plumbing and the metadata-only roundtrip unit tests. The issue is an RFC seeking agreement on lifecycle, consistency, and compatibility; done means a maintainer decision on the direction rather than a narrowly defined patch.

Written by the indexing model from the issue text.