nvsnap: e2e can measure a cold start and report it as a restore

The restore step does not verify that the pod it launched was actually admitted as a restore. When it was not, the pod cold-starts, serves normally, passes every downstream check, and the run reports restore timings that are really cold-start timings.

How it happens

Rootfs/cachedir restore manifests carry a placeholder:

nvsnap.io/restore-from: "__CAPTURE_HASH__"

test-e2e.sh substitutes it. Anything that applies the template without substituting -- a probe script, a manual kubectl apply, a future code path that skips the sed -- leaves the placeholder in place. The webhook has no hash to resolve, so it injects no cache mount and no cache env. The pod then starts cold.

Nothing downstream notices. The pod becomes Ready, /v1/models answers, inference succeeds, and the harness prints a restore time.

Observed

A 70B TP=4 "restore" measured this way spent 8m47s downloading model weights, because the pod had no captured cache mounted at all:

Time spent downloading weights for meta-llama/Llama-3.1-70B-Instruct: 527.28 seconds

Pod spec confirmed the cause -- no /opt/nvsnap mount, and HF_HOME=/root/.cache/huggingface rather than the stamped cache path. The resulting numbers were used to reason about restore performance before the mistake was caught.

Fix

Two guards in test-e2e.sh:

1. Refuse to apply a restore manifest that still contains __PLACEHOLDER__ text.
2. After creating the restore pod, assert the webhook decorated it: the captured cache is mounted under /opt/nvsnap, and cache env vars (HF_HOME, NIM_CACHE_PATH) point into it. Fail loudly otherwise.

Both were verified against the saved spec of the pod that produced the bad number.

Why it matters beyond the harness

This is the same shape as two other issues found this week: a check that reports success while measuring or recording the wrong thing. A restore benchmark that silently degrades to a cold start is worse than a failing one, because the number looks plausible and gets quoted.

Start in test-e2e.sh and inspect how restore manifests are substituted and applied. Add the two restore-validity checks described in the issue, then run the end-to-end restore flow and confirm it fails when the placeholder remains or the pod lacks the /opt/nvsnap mount and cache environment paths.