Checkpoint DELETE drops the catalog row even when the L1 cascade fails, orphaning the dump

Found on nvcf-dgxc-k8s-aws-usw2-dev2 while cleaning up after auth testing. Independent of #555, though auth is what exposed it.

Behaviour

cascadeDeleteCheckpoint deletes the catalog row even when the L1 tier delete fails. The API returns 204 and the caller believes the checkpoint is gone, while the on-disk dump survives with nothing left pointing at it.

Observed:

DELETE /api/v1/checkpoints/247eb2e0...__20260809-194514  ->  204
catalog: 0 rows
disk:    33G  /var/lib/nvsnap/checkpoints   (unchanged)

Why the orphan is unrecoverable

The catalog row is the only record tying a checkpoint id to its L1 location. Once dropped:

• checkpoint.sh cleanup cannot find it, since it enumerates through /api/v1/checkpoints.
• DELETE by id returns 404; there is no row to cascade from.
• No periodic sweep reclaims it (see #206, still open, for the equivalent blobstore gap).

The only remaining recourse is removing the directory on the node by hand, which is what I had to do.

Impact

Every failed cascade permanently leaks a full checkpoint. These are large: a single vllm-small dump on this cluster is 33G, of which 32G is pages-11.img. A handful of failures fills a node disk, and the only signal is a warning in the agent log.

The failure mode is not hypothetical. It fired for every delete while the agent ran with --auth-mode=required, because nvsnap-server does not send the agent token (details in #555).

Suggested fix

Do not delete the catalog row unless the L1 delete succeeded. On partial failure, keep the row and surface the state, so the checkpoint stays enumerable and a retry can finish the job. Returning 207 or 500 with a machine-readable per-tier result would let callers distinguish "gone" from "partially gone"; 204 currently claims more than the server knows.

A reconcile sweep that reclaims dumps with no catalog row would be defence in depth, and would pair naturally with #206.

Related

• #555 auth is what made the cascade fail here
• #206 periodic orphan GC in the blobstore, the same class of gap one tier down