Holdout-masked builds + population-view scorecards as release gates (incl. pre/post-calibration delta)

Proposal

Design per-survey holdouts into the US build so that certified bundles can be scored through the population-view harness non-self-referentially, and adopt the harness scorecards as release gates.

1. Holdout masking at the source lane. Before the support pool and any donor fitting, reserve a seeded holdout from each source survey (ASEC vintages, SCF, later SIPP). Holdout records appear nowhere downstream: not in the support spine, not in donor fits, not in calibration targets derived from the sources.
2. Multi-view scorecards as promotion gates. Score each candidate bundle against every survey's holdout through its view (that survey's variables, its weights): weighted energy distance (strictly proper), PRDC coverage (support-based, invariant to reweighting of the candidate — the calibration-blind block), weighted classifier two-sample AUC, and the uncapped per-variable tail block (q90/q99 ratios, W1/sd). Anchor each view with a sampling floor (the survey's own complementary split scored as a candidate). Promotion requires no regression vs the incumbent on all blocks, tail ratios within floor-calibrated bands.
3. Pre/post-calibration delta as standard QA. Run the harness on the same records before and after calibration: coverage is invariant to reweighting by construction, so the delta isolates what calibration did to the weighted joint — a diagnostic that does not exist today (calibration QA checks target attainment and weight statistics only).

Why (measured)

The imputation paper (github.com/PolicyEngine/imputation-paper) built and validated exactly this machinery on a two-view SCF→CPS instantiation: floors behave (energy 0.009–0.012, AUC ~0.50), the block decomposition localizes failures (a weight-blind fit posts the highest coverage while its q99 runs 1.9–2.25× — measure wrong, support right, hence repairable by reweighting; an OLS-style fit collapses support, which no reweighting repairs), and richer conditioning makes joint failures decisively detectable (marginal-draw C2ST 0.85→0.97 from 6 to 10 shared predictors). Today's certified bundles cannot be scored this way because their sources overlap any holdout — the masking has to be built in, not bolted on.

Reference implementation: experiments/views.py + experiments/metrics.py + cli/harness.py in the paper repo (floors, view projection, all four blocks, manifests). Related: #296 (ASEC vintages), #297 (auxiliary predictors), populace-benchmarks#6 (tail/fragility promotion metrics — the single-file version of these gates).

🤖 Generated with Claude Code

Start by reviewing the referenced experiments/views.py, experiments/metrics.py, and cli/harness.py implementation in the imputation-paper repository, then read related issues #296, #297, and populace-benchmarks#6. Map the proposed holdout masking, multi-view scorecards, and pre/post-calibration delta into microcosm; done means these checks can act as promotion gates for certified bundles.