Promotion certification suite: pre-registered out-of-sample scorecard at default-flip time

Promotion certification suite: pre-registered out-of-sample scorecard at default-flip time

Problem

Datasets become the policyengine.py default via a pin-bump PR whose evidence is currently assembled ad hoc (e.g. #427's matched-N comparison for the UK, per-build gate reports for the US). Three gaps:

1. Authority location. Backtests computed inside populace at build time are diagnostics (right — see populace#348); nothing with pass/fail authority runs at the promotion boundary, where the question is "is this file fit to be the default under the engine users will actually run".
2. Engine entanglement. Build-time scorecards are stamped under the build's engine pin; policyengine.py ships newer engines, so those scorecards go stale. Reform deltas vs external scores test dataset × engine jointly and only make sense evaluated with the shipping engine.
3. Goodhart containment. Without pre-registration and a ledger of evaluated candidates, iterating builds against held-out metrics quietly converts them into targets.

Proposal

A certification runner in policyengine.py that the default-flip PR must carry:

• Inputs: candidate dataset URI (sha-pinned), the engine version the bundle will resolve, a pre-registered metric set + bands (frozen in-repo before the candidate exists; changes to the set are PRs with their own review).
• Output: one scorecard artifact attached to the pin-bump PR, plus a ledger line recording every candidate evaluated (so "N builds were tried before one passed" is visible).
• Metric families (per country): population-level backtests never used as calibration targets — poverty rates vs official statistics, program caseloads/expenditures vs administrative totals, reform deltas vs external scores (JCT/CBO; OBR/HMRC) — plus mechanical fitness checks (column-manifest completeness per populace#340, weight-health summary).
• Relationship to populace: populace keeps build-integrity gates + freely consulted diagnostics (populace#348, #302); the promotion suite is evaluated once per candidate at the boundary. Fix path for a failed metric is inputs/joint structure in populace, never fitting the output.

First two instances

US — next default flip (populace#299 Build H candidate). Scorecard: SPM poverty (national + state, Census P60-287, the populace#348 rows), SNAP/SSI/Medicaid caseload-and-cost vs administrative totals, OBBBA reform deltas vs JCT scores (with the #340 absent-column families explicitly reported), income tax / Social Security aggregates vs SOI/SSA under the shipping engine.

UK — retroactive adjudication of populace-uk vs enhanced FRS. The #427 promotion rested on a sound matched-N holdout comparison (6/6 rotations) — strong but scoped to the calibration surface, and per-target it was 79–70. The suite upgrade: popdgp joint-distribution metrics (energy, C2ST) plus out-of-sample backtests — HBAI poverty rates (overall/child/pensioner, BHC/AHC), DWP benefit caseloads and expenditure (UC, state pension, PIP), HMRC income-tax liabilities by band — run identically over both artifacts under policyengine-uk at the bundle pin. This either upgrades "wins the measured surface" to an unqualified claim we can publish, or finds where the incumbent still wins (e.g. the private-school-students-style targets). The transfer-paper eval-pack machinery (pre-registered config, frozen shas, scorecard-only output) is the template and makes this cheap.

Non-goals

• Not a release gate inside populace builds (selection-pressure containment; populace#302 covers holdout-masked build gates separately).
• Not continuous re-certification on every engine bump initially — start with flip-time evaluation; scheduled re-runs can come later.

Refs: populace#348 (poverty backtests as diagnostics), populace#302 (holdout-masked build gates), populace#305 (validation-portfolio META), populace#340 (column manifest), #427 (UK promotion evidence), populace#8 (UK sound comparison).

Start by reading policyengine.py's pin-bump and promotion flow, then review the transfer-paper eval-pack machinery described as the template. Define how the runner accepts frozen candidate inputs and produces the scorecard and ledger entry. Done means a candidate can be evaluated once at the promotion boundary with pre-registered metrics and a reproducible scorecard artifact.