NVIDIA / NVIDIA/nvcf

nvsnap: L2 promote completes but pvc_promote_state is never written (blocks NVCA warm gating)

Open
#469 1 comment 0 reactions 1 assignee View on GitHub

@balajinvda is already working on this.

Since Jul 27, 2026.

nvsnap
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Summary

L2 promote succeeds and creates the shared read-only artifact, but the catalog
never records it. pvc_promote_state stays empty, so the restore-side resolver
believes there is no L2 tier and falls back to a node-local overlay. The
promoted volume is created, paid for, and never used.

Reproduced on the cachedir/rootfs capture path with an NVMesh L2 StorageClass
(shared-volume / zero-copy strategy).

Root cause

maybeMarkPVCPromoteReady (internal/server/server.go) is a one-shot check run
once at end-of-capture. L2 promote is deliberately asynchronous. The check
therefore runs while promote is still in flight, finds no rox PVC, and returns
a no-op at Debug level. Nothing re-checks afterwards.

Observed ordering on a 14 GB capture:

~23:02:00  capture finishes; maybeMarkPVCPromoteReady -> rox PVC NotFound -> no-op
 23:04:32  agent-direct copy complete   bytes=14253175144 files=315
 23:04:43  L2 promote complete (shared-volume, zero copy)  rox=rox-<hash>
           catalog pvc_promote_state = ""     (never written)

The race window is the whole promote duration, so this loses for any capture
large enough to matter.

The agent's own state write cannot cover the gap: on this path Backend.Put
runs before the catalog row has its hash, so the agent's write returns
hash-not-found and is intentionally swallowed. setState's comment names the
server as the component responsible for the final write, and the server only
tries once.

Impact

  • L2 fan-out has never actually been exercised end to end; restores silently
    use the fallback path.
  • Measured cost on a 7B embedding model: restore total 90.0s with model
    download 61.8s, against a 106.0s cold start. Almost the entire benefit of L2
    is lost.
  • NVCA gates warm state on pvc_promote_state=ready. On this path that value
    never arrives, so a function would stay cold indefinitely.
  • The failure is silent by construction: NotFound is expected for genuine
    no-L2 clusters, so it is logged at Debug and looks identical to success.

Direction

The component that knows promote finished should record it. Options:

  1. Agent-side write on promote completion, retrying until the catalog row has
    its hash rather than swallowing the first hash-not-found permanently. Most
    direct: no polling, and the writer is the one holding the result.
  2. Server-side retry/watch on the rox PVC until it appears or a deadline
    passes, instead of a single lookup.

Whichever is chosen, the silent-Debug no-op should be distinguishable from the
genuine no-L2 case, so a stuck promote is visible rather than looking like a
cluster without L2 configured.

Verification

A capture is only correct when the catalog row reaches
pvc_promote_state=ready with rox_name set, and a subsequent restore mounts
the shared claim rather than re-downloading the model. Restore model-download
time near zero is the observable signal; the current fallback shows it at
roughly cold-start cost.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.