PolicyEngine / PolicyEngine/microcosm
One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Where this sits
#578's acceptance rule ships a candidate iff it beats the incumbent on the frozen comparison register and the invariant battery passes — and it specifies the battery as "versioned, machine-checked in one batched pass, and recorded in both manifests."
That sentence is currently true in one place and approximated in two others. This issue makes it true everywhere, by promoting the batched-report shape the UK path already has into country-agnostic tooling that the UK, the US, and Belgium all consume.
Increment 1 is the shared module with the UK as its first consumer. The UK migration is behaviour-preserving by construction, so it is reviewable as a refactor.
The three mechanisms today
| surface | artifact | versioned | aggregate verdict | attested | membership checked |
|---|---|---|---|---|---|
UK national — uk_runtime/terminal_gates.py |
terminal_gates.json, schema 3 / attestation 5 |
yes | yes | HMAC-SHA256 | yes |
US national — tools/build_us_fiscal_refresh_release.py |
per-gate keys in the build block of calibration_diagnostics.json |
no | no | no | no |
US local-area — tools/build_us_acs_local_release.py |
gate_summary.json, verified by _check_local_area_gates (contract.py:2773) |
no | per-gate only | no | no |
The third retires with the acs_local product under #578's scope hardening. The first two must converge.
The asymmetry is not gate count. The US has far more checks: 66 distinct gate result objects across the release build, 12 of them threaded into _release_gate_failures (build_us_fiscal_refresh_release.py:6312-6329), plus 4 pre-solve checks in release_gate_preflight.run_preflight() and the pool builder's terminal agreement gate. The difference is that a US gate's identity lives in three hand-maintained places — the local variable, the prose prefix in the failure collector, the manifest key in the writer — so nothing structurally forces consolidation and the surface accreted. The UK has the better container with fewer things in it. Neither side should win: the UK's container plus the US's per-evidence placement is strictly better than either.
The seam already exists, unused
packages/populace-build/src/populace/build/country_spec.py already defines GateSelectionSpec(id, gate, criticality, parameters, notes) and GatesManifest, with an ALLOWED_GATE_FUNCTIONS allowlist, duplicate-id rejection, and a rule that a country with only diagnostic gates has no release contract. Belgium has a complete spec — build/be/gates.json, 9 gates across both criticality tiers — and test_country_spec.py:122 already asserts that BE deselects parity and export_surface because it has no incumbent. Per-country gate applicability is therefore already designed and golden-tested.
What is missing: nothing anywhere loads a spec and runs its gates. Selection is purely declarative. build/uk/ and build/us/ carry only country_package.json. ALLOWED_GEOGRAPHY_SPINE_METHODS is documented in-file as "the UK OA pattern, generalized" — the module was written country-agnostic and has been waiting for a second consumer.
End state
One executor, one report format, one verifier contract, three countries. Every gate declares the phase it runs in and whether it blocks. The report is written unconditionally at every phase boundary. Publication is gated on the report rather than on scattered if not gate.passed branches.
Increment 1 — country-agnostic tooling, UK as first consumer
1. country_spec gains phase, and the allowlist widens
Add phase: str to GateSelectionSpec, validated against a new ALLOWED_GATE_PHASES seeded with the phases that already exist: preflight, assembled, transferred, simulated, terminal. First and last cover the UK as it stands; the middle three are the pool builder's existing POOL_CHECKPOINT_STAGE_ORDER.
Extend ALLOWED_GATE_FUNCTIONS (14 entries, BE-shaped) with the names the UK battery needs — release_input_coverage, degenerate_release_surface, zero_weight_strata, weight_ess, weight_ratio, weights_audit, target_fit, input_mass_parity, tail_concentration — plus spine_agreement for the pool builder. The shared vocabulary stays country-neutral (weight_ess, not uk_weight_ess); UK wrappers keep their internal names.
The precedent to copy is in the repo: SourceOperatorContract (us_runtime/multispine_pool.py:324-330) carries a phases tuple, with per-phase orders derived by filtering at :462-474. Same shape, applied to gates instead of operators.
2. A shared executor
populace.build.gate_battery.evaluate_gate_battery(spec, phase, evidence, implementations)
-> GateBatteryReport
Behaviour lifted from uk_terminal_gate_report(), which already does all of it correctly:
- build
(name, callable)evaluators from the spec entries for that phase; - evaluate every one even when an earlier fails, converting per-gate exceptions into failed
GateResults — reuse_evaluate_gate(terminal_gates.py:1262-1298) verbatim, including its check that the returnedGateResult.namematches the requested name; - never raise. Return the report; the caller decides to block.
Batched within a phase, fail-closed at the phase boundary. That combination is what lets a gate sit where its evidence appears without giving up the single batched report.
3. Outcome taxonomy
Blocking makes the report necessarily partial, so a boolean per gate is not enough. Each registry entry resolves to exactly one of:
passed · failed · not_applicable · evidence_absent · unreached
not_applicablerequires a declared reason, reusing the reviewed-exclusion discipline already implemented inuk_runtime/weighted_integrity.py(mandatory reason, dormant entries reported, stale entries fail). UK take-up becomesnot_applicablewith reason "no take-up assignment surface; #578 item 5" — a receipt rather than a silence.unreachedmeans an upstream phase blocked first. Conflating it withnot_applicableis the failure mode this battery exists to prevent, so the report carriesphases_evaluatedandblocked_at_phase, and the attestation covers both.- Shipping requires every
release_blockingentry to bepassedornot_applicable.diagnosticentries are recorded and never block.
4. Write-then-block, codified
Already the de facto behaviour in all three places, held up by convention rather than construction:
- UK —
national_build.py:427-442evaluates,:443-451writes the report,:452-455raises,:457writes the H5. Report persisted before the raise; H5 never written on failure. - US national —
_write_release_calibration_diagnostics(:6856-7066, docstring "Write calibration diagnostics even when hard release gates fail.") is called at:10631with a possibly non-empty failure list; the H5 write at:10995is guarded by the check at:10955. - US pool builder — agreement diagnostics are written on failure via
GateReport((result.agreement_gate,)).to_manifest()(build_us_multispine_pool.py:1913) and the manifest is marked non-ready, but the pool H5 is still written, because it is an intermediate rather than a publication.
So the executor needs two blocking modes: blocks_artifact (UK and US release — nothing is written) and marks_artifact (pool builder — artifact written, manifest records the verdict). Both write the report first, and the ordering gets a test that fails a gate and asserts the report is on disk.
5. Verifier: one pinned spec fingerprint per country
contract.py cannot import populace-build. It would be a dependency cycle — populace-build[us] already depends on populace-data, and populace-data deliberately declares only huggingface_hub, h5py, packaging so that loading a published population does not pull the build toolchain. It would also make the verifier derive its expectations from the thing it verifies: a PR loosening a threshold would loosen the check meant to catch it.
So the release ships its spec digest, and contract.py pins one expected spec.fingerprint per country and validates the report against the shipped spec. Deleting a gate moves the fingerprint, the pin mismatches, and publication fails until someone reviews the pin move — the same doctrine as the existing policy pin ("a threshold outside this hash is not attested"), applied to membership instead of thresholds.
CountrySpec.fingerprint (country_spec.py:616-624) already computes this as a sorted-then-hashed composition over resource_hashes, and the golden test already asserts stability across loads. This replaces the hand-mirrored _UK_ALWAYS_APPLICABLE_GATE_NAMES and _UK_TERMINAL_EVIDENCE_GATE_NAMES tuples, and stays flat as countries are added rather than growing with countries × gates.
6. UK migration is behaviour-preserving
Author build/uk/gates.json from the current 11-gate battery: the 5 always-applicable gates (release_input_coverage, degenerate_release_surface, zero_weight_strata, weight_ess, weight_ratio), weights_audit, the parity trio (export_surface, target_surface, target_fit), and #609's input_mass_parity and tail_concentration. All at phase: terminal; the two cheap assertions at national_build.py:416-420 become phase: preflight.
The UK is already exactly phases = (preflight, terminal), so declaring them changes nothing about when gates run. The PR asserts identical per-gate verdicts against the committed June fixtures.
7. Belgian compatibility, proved rather than asserted
The executor must load and run build/be/gates.json as it stands. BE has no build yet, so the test is: load the spec, resolve all 9 gate names, and assert a well-formed report over synthetic evidence. A BE gate name with no implementation is a named gap in the report (evidence_absent), not a crash. This is what stops the BE onboarding from becoming a third parallel battery.
Later increments
- US pool builder — adopt the executor at the
assembled/transferred/simulatedboundaries, which are already checkpoint boundaries. Smallest US surface, already usesGateReport, and the one where phase-boundary blocking saves money (run-5 postmortem: four preemptions, zero completions, ~$122). - US national release — migrate the 12 blocking gates off the hand-threaded kwargs in
_release_gate_failures, then the remaining 54. The per-gate keys incalibration_diagnostics.jsonare read only by tests, so this is a test rewrite rather than a consumer break. It also adds a required gate-report file to the US contract:required_release_files()(contract.py:275-281) currently returns no gate report for US releases. gate_summary.jsonretirement alongside theacs_localproduct.- BE onboarding consumes the module directly and adds no new battery.
Settled decisions
- Report file: reuse
terminal_gates.json. The filename is already country-neutral; onlycontract.py's constant is named_UK_*. The shape change bumps schema 3→4 and attestation 5→6, and those pins become a per-country lookup. The published June release is immutable and can never carryphaseor outcome fields, so it keeps validating through the_UK_LEGACY_RELEASE_IDSbranch that already exists atcontract.py:761for exactly this case. #588 performed that migration once, so the shape is known. US and BE then emit the same file rather than a fourth format. - Membership anchor: pinned
spec.fingerprintper country, not a mirrored gate list. - Scope: the end state is recorded here; increment 1 delivers the tooling with the UK as first consumer and BE proved compatible.
Non-goals
- No new gates. This issue moves existing gates into shared scaffolding; new checks arrive on their own issues (#609 for the weighted-integrity pair, #578 items 5–8 for the rest).
- No threshold changes. Every country keeps its own adjudicated constants, pinned in its own policy hash.
- No loosening for convenience: a gate that cannot run is
evidence_absentor a reasonednot_applicable, never an omission.
Relates
- #578 (the acceptance rule this implements; UK backlog items 1–8)
- #609 (increment 4's two gates are the first migration payload)
- #588 (the batched attested report being generalized)
- #263 (BE calibration surface — the second consumer)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/populace-build/src/populace/build/country_spec.py, test_country_spec.py, and the existing UK flow in uk_runtime/terminal_gates.py and national_build.py. Trace uk_terminal_gate_report() and _evaluate_gate(), then confirm the work is complete when UK behavior remains unchanged, Belgium produces a well-formed report, and report-before-blocking is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, testing, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100