Make the US release build transactional: stable-name H5 lands before late gates, the solve has no checkpoint, and release-ID reuse manifest-lists stale files

Summary

The US release build writes into its final locations as it goes and persists its most expensive product last. A late gate failure (the normal failure mode — smoke and coverage run after export) leaves a usable-looking H5 under the stable artifact name with no completion marker, and throws away a multi-hour solve. Release-ID reuse compounds it: an existing nonempty release directory is accepted, and stale optional files from a previous attempt are silently manifest-listed.

Evidence (current main, 8264c49)

In-place construction. tools/build_us_fiscal_refresh_release.py:5645-5656: artifact_root and releases/<id>/ are created with mkdir(parents=True, exist_ok=True) at startup; diagnostics are written directly into them throughout the run.

The H5 lands on its stable name before the late gates. :6510-6511 writes artifact_root / DATASET_FILENAME (populace_us_2024.h5 — the same stable filename the publisher uploads and operators inspect) and only then runs the reform-coverage smoke (:6521-6570), the optional export audit, reform validation, demographics, and the final take-up check. Any of those failing leaves the new uncertified H5 in place under the certified name.

The calibration NPZ is written after the H5 and the smoke. :6588-6590. The solve (selection ~1,500 epochs + refit ~1,500 epochs, hours of wall time) has no optimizer checkpoint of its own — so a smoke failure discards the entire optimizer result even though nothing about the weights was wrong. Restart support today covers only the target frame and per-reform vectors (#317).

Release-ID reuse accepts stale state. --release-id reuses an existing directory (exist_ok=True, no emptiness check), and _build_manifests (:5385-5407) includes reform_validation.json / demographics.json in the release manifest merely if the files exist — a rerun with --skip-reform-validation in a reused directory manifest-lists the previous attempt's stale file with a fresh SHA, as if this build produced it.

Manifests are separate direct writes. build_manifest.json and release_manifest.json are two independent write_text calls (:5239-5243, :5411-5413) — no temp-file + os.replace, no atomic bundle promotion (the pattern the checkpoint writer itself already uses at :1258-1288).

A known config contradiction only fails after target materialization. --warm-start-calibration-npz on the default sparse path is always fatal: main() forwards warm_start_weights into calibrate_l0_refit (:6151-6170), which forwards them into the L0 selection solve, which rejects them (packages/populace-calibrate/src/populace/calibrate/solve.py:1441-1447). The error costs a full target materialization; it should be an argparse-time preflight (dense runs are the only valid consumers of that flag today).

Fix sketch

1. Build into a temporary run directory; on success, validate the completed bundle and atomically promote (rename) into releases/<id>/ + artifacts/. The stable-name H5 appears only after every gate has passed.
2. Write the calibration NPZ immediately after the solve returns, before export — it is the restart point for everything downstream.
3. Refuse a nonempty existing release directory unless an explicit --resume is passed; under --resume, refuse to manifest-list files this run did not produce or verify.
4. Reject --warm-start-calibration-npz with the sparse default path at argument-parse time.

Acceptance criteria

• A build killed (or gate-failed) after export leaves no new file under the stable artifact names.
• A smoke-gate failure can be retried without re-running the solve (NPZ persisted at solve completion).
• Rerunning with an already-used --release-id without --resume is an error; with --resume, stale optional files are not manifest-listed.
• --warm-start-calibration-npz without --dense-default-dataset fails before any data is downloaded.

Cross-links

• #299 (Build E OOM/restart pain — the operational cost of weak resumability)
• #217 / #317 (target-frame and reform-vector restartability — this extends the same principle to the solve and the bundle)
• #368 (Build J will run this exact late-gate sequence; smoke is expected RED until Deliverable 2 lands, which makes solve-loss-on-smoke-failure a live cost)
• #346 (sparse warm-start design — the eventual legitimate consumer of warm starts on the sparse path)

Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.

Start in tools/build_us_fiscal_refresh_release.py at main(), the export and late-gate sequence around lines 6510-6590, and _build_manifests around 5385-5413. Read the checkpoint writer at lines 1258-1288 and the warm-start validation path in packages/populace-calibrate/src/populace/calibrate/solve.py:1441-1447. Done means failed or killed runs do not publish stable artifacts, solve output is restartable, release reuse is guarded, and invalid sparse warm starts fail before download.