hardening: truncated cache blob poisons the workspace (atomic store + clean-miss fetch)

Symptom (hit in prod)

A job failed at PREPARE→RUN because a cache blob was truncated, and the failure poisoned the workspace:

cache "gradle-deps-…": fetch failed (exec untar in cache-fetch:
  sh -c set -o pipefail; zstd -dc -T0 | tar -xf - -C "$1"
  command terminated with exit code 1
  stderr="/*stdin*\ : Read error (39) : premature end \ntar: short read\n") — task runs without cache
...
Exception in thread "main" java.lang.ClassNotFoundException: org.gradle.launcher.GradleMain
── task failed after <1s (exit 1)

Root cause

Two independent defects compound:

1. The stored cache blob is truncated (incomplete store). zstd: premature end (39) means the zstd frame ended mid-stream — the blob was never fully written. The tar: short read is only the downstream consequence. Codec detection worked correctly (it read the magic bytes and picked zstd; the blob genuinely was zstd). The likely trigger is a cache store interrupted mid-upload (spot preemption / crash / network) that still publishes the partial object — the store is not atomic/verified.

2. Cache-fetch streams zstd -dc | tar -xf straight into the workspace, so a truncated blob leaves a PARTIAL extraction behind. The isolated (in-pod) restore extracts in streaming fashion; on truncation, the files that arrived before the error stay on disk. Here the partially-extracted Gradle distribution was incomplete → the wrapper found the dist dir but gradle-launcher-*.jar was missing → ClassNotFoundException: org.gradle.launcher.GradleMain. "task runs without cache" does NOT undo the partial extraction, and because the bad blob lives in the bucket, every subsequent run refetches it and fails the same way (recurs until the blob is purged). Confirmed: purging the cache key via the UI unblocked it immediately.

Not related to artifact write-once (#210): caches are never signed create-only (RequestCachePut → SignedPutURL without WithCreateOnly). Verified.

Proposed fixes

1. Atomic / verified cache store. Write to a temp/generation key and promote only after the full upload + a sha check succeeds (or verify sha server-side on confirm before marking the row ready), so an interrupted store never publishes a truncated blob. Overlaps the cache-integrity direction in #298 — could be folded in or kept as its own slice.
2. Fetch must fail CLEAN. The isolated restore should extract to a temp dir and mv into the workspace only on success (or verify the blob's sha before extracting), so a truncated/corrupt blob becomes a clean cache miss instead of a poisoned workspace. The agent Go path already sha-checks via DownloadAndUntar; the in-pod zstd -dc | tar -xf path needs the same guarantee.
3. (Optional) Self-heal a known-bad blob. On a fetch decompress/untar failure, mark that cache row/blob for eviction so the next run re-stores instead of refetching the corrupt object.

Acceptance criteria

• An interrupted cache store (killed mid-upload) never leaves a fetchable truncated blob (store is atomic or the row is only marked ready after a verified complete upload).
• A corrupt/truncated cache blob results in a clean cache miss (job proceeds and re-stores), never a partially-extracted workspace.
• Test: a deliberately truncated cache blob → fetch → job still succeeds (miss path), workspace has no partial extraction.

Context

Real incident: LOAN integration job, backend GCS (cora-gocdnext), zstd cache codec (#274). Related: #298 (cache write race — old signed URL clobbers newer blob). Both are the cache-write-integrity class.

Start by tracing RequestCachePut and SignedPutURL for the store path, then compare the in-pod zstd/tar restore with the agent's DownloadAndUntar path. Reproduce a truncated cache blob and an interrupted upload, and inspect the cache-write and restore tests. Done means corrupt data produces a clean miss with no partial workspace and incomplete stores are not fetchable.