nvsnap: same-node warm tier (tier-0 frozen donor / shared weight cache) for near-instant same-node restart

Summary

Design a same-node warm tier for nvsnap: a tier-0 hot layer above the existing
cross-node tiers (peer cascade / shared-volume L2 / blobstore) that keeps a
model resident on the node so a same-node restart or scale-up is near-instant.
This is explicitly same-node only; it complements the cross-node fan-out path,
it does not replace it.

Motivation

Cross-node restore is served by the shared-volume and cascade tiers. The
remaining floor on those paths is the per-process CUDA context re-creation
(module/kernel reload, the cuInit cost), which storage speedups cannot remove.
For workloads whose recovery pattern is same-node (crash recovery, rolling
restart, same-node autoscale burst), a resident warm donor removes that floor
entirely.

External corroboration: SGLang is adding a persistent weight-cache daemon
(sgl-project/sglang#27139) and NVIDIA GMS holds weights resident via the CUDA
VMM API; both target the same same-node fast-restart problem.

Two flavors

A. Frozen donor (full checkpoint resident)

Keep a checkpointed process live and frozen on the node: CUDA context, compiled
kernels, captured graphs, and weights all resident. A same-node restore thaws or
clones the donor instead of cold-restoring. Skips the entire restore path,
including the CUDA context re-creation floor. This is the differentiated option
because nvsnap already has the frozen-process machinery.

• Pro: fastest possible restart; nothing re-materialized.
• Con: pins a full GPU context + the model's VRAM per held donor.

B. Shared weight cache

A persistent process holds only the weights resident in VRAM and maps them into
restarting engines via CUDA IPC. Cheaper than a full donor, and it has a second
payoff: multiple same-node replicas can share one weight copy instead of each
holding its own, so it is also a VRAM-density win, not only a latency win.

• Pro: lower VRAM cost; enables replica packing per node.
• Con: skips weight load only; context/graph re-init still runs. Overlaps
  SGLang/GMS work, so integrate rather than reinvent where possible.

Open design decisions

• Which flavor first (A is the nvsnap differentiator; B is cheaper and denser).
• Single-consumer thaw vs multi-consumer clone of a donor.
• VRAM budget: how many hot models may be held resident per node.
• Eviction policy when GPU memory is contended.
• Interaction with the existing tiers: warm tier is tried first on same-node
  restore, falling through to L1/L2 when no donor is resident.

Non-goals

• Cross-node restore (served by the existing shared-volume and cascade tiers;
  CUDA IPC and a resident donor are same-node by nature).
• Replacing the cross-node fan-out path.

References

• sgl-project/sglang#27139 (weight-cache daemon: persistent VRAM weights via
  CUDA IPC)
• NVIDIA/nvcf#389 (injected cooperative checkpoint/restore hooks)