nvsnap: L2 promote completes but pvc_promote_state is never written (blocks NVCA warm gating)
@balajinvda is already working on this.
Since Jul 27, 2026.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Summary
L2 promote succeeds and creates the shared read-only artifact, but the catalog
never records it. pvc_promote_state stays empty, so the restore-side resolver
believes there is no L2 tier and falls back to a node-local overlay. The
promoted volume is created, paid for, and never used.
Reproduced on the cachedir/rootfs capture path with an NVMesh L2 StorageClass
(shared-volume / zero-copy strategy).
Root cause
maybeMarkPVCPromoteReady (internal/server/server.go) is a one-shot check run
once at end-of-capture. L2 promote is deliberately asynchronous. The check
therefore runs while promote is still in flight, finds no rox PVC, and returns
a no-op at Debug level. Nothing re-checks afterwards.
Observed ordering on a 14 GB capture:
~23:02:00 capture finishes; maybeMarkPVCPromoteReady -> rox PVC NotFound -> no-op
23:04:32 agent-direct copy complete bytes=14253175144 files=315
23:04:43 L2 promote complete (shared-volume, zero copy) rox=rox-<hash>
catalog pvc_promote_state = "" (never written)
The race window is the whole promote duration, so this loses for any capture
large enough to matter.
The agent's own state write cannot cover the gap: on this path Backend.Put
runs before the catalog row has its hash, so the agent's write returns
hash-not-found and is intentionally swallowed. setState's comment names the
server as the component responsible for the final write, and the server only
tries once.
Impact
- L2 fan-out has never actually been exercised end to end; restores silently
use the fallback path. - Measured cost on a 7B embedding model: restore total 90.0s with model
download 61.8s, against a 106.0s cold start. Almost the entire benefit of L2
is lost. - NVCA gates warm state on
pvc_promote_state=ready. On this path that value
never arrives, so a function would stay cold indefinitely. - The failure is silent by construction: NotFound is expected for genuine
no-L2 clusters, so it is logged at Debug and looks identical to success.
Direction
The component that knows promote finished should record it. Options:
- Agent-side write on promote completion, retrying until the catalog row has
its hash rather than swallowing the first hash-not-found permanently. Most
direct: no polling, and the writer is the one holding the result. - Server-side retry/watch on the rox PVC until it appears or a deadline
passes, instead of a single lookup.
Whichever is chosen, the silent-Debug no-op should be distinguishable from the
genuine no-L2 case, so a stuck promote is visible rather than looking like a
cluster without L2 configured.
Verification
A capture is only correct when the catalog row reaches
pvc_promote_state=ready with rox_name set, and a subsequent restore mounts
the shared claim rather than re-downloading the model. Restore model-download
time near zero is the observable signal; the current fallback shows it at
roughly cold-start cost.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.