PolicyEngine / PolicyEngine/microcosm

Bind every blocking gate verdict into release certification: waived gates vanish from manifests and publication accepts any nonempty gates mapping

Open
#381 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Summary

Gate execution (what blocks a build) and gate certification (what the published evidence proves) are different contracts. A release can pass publication while blocking gates were waived, skipped, or never represented in the manifest at all — and the publication contract cannot tell. #369 made input coverage a hard gate inside the build; this issue is about binding every blocking gate's verdict into the certification surface so the published artifact carries the proof.

Evidence (current main, 8264c49)

Waived gates vanish from the manifest. tools/build_us_fiscal_refresh_release.py:6305-6310:

enforced_input_mass_reference_gate = (
    None if args.allow_input_mass_drift else input_mass_reference_gate
)
enforced_ecps_parity_gate = (
    None if args.allow_ecps_parity_gaps else ecps_parity_gate
)

These Nones are what _build_manifests receives (:6726-6728) — a waived mass or parity gate leaves no entry in build_manifest.gates, indistinguishable from a build where the gate never existed.

Half the blocking gates are not accepted by _build_manifests at all. Its signature (:5008-5035) takes twelve gates (target profile, health, base population, immigration, mass, degenerate, parity, hours, SNAP take-up, eligibility, pregnancy, SNAP exemptions) — but not: the take-up signal gate (:5783), the SCF-wealth gate (:5962), the input-coverage gate (#369, :6396), the reform-coverage smoke (:6521), or the Medicaid/ACA stage gates. The coverage and smoke gates write side files (input_coverage.json, reform_coverage_smoke.json, with an honest "enforced": bool) — but those files are neither in release_manifest.json's artifact list (:5360-5409) nor required by the publication contract.

The publication contract accepts any nonempty gates mapping. packages/populace-data/src/populace/data/contract.py:45-52 requires only build_manifest.json, release_manifest.json, calibration_diagnostics.json (+ US source coverage); :424-428 checks only isinstance(manifest.get("gates"), Mapping). Deleting input_coverage.json from a release directory — or shipping one with passed: false, enforced: false — does not fail validate_release_dir. The contract's own valid-release fixture certifies with a single arbitrary gate (test_contract.py).

Engine-drift assertions for the take-up contract are test-only. assert_take_up_contract_current / assert_take_up_treatments_consistent (packages/populace-build/src/populace/build/us_runtime/take_up_contract.py:284,351) have no call site outside test_us_take_up_contract.py — an engine bump that changes a take-up default or formula treatment between CI and a release run is not checked at build time.

Reform probes cannot fail on direction. us_reform_coverage_smoke_gate computes a signed effect (probes declare effect_direction) but passes on abs(effect) >= min_abs_effect (packages/populace-build/src/populace/build/us_runtime/reform_coverage_smoke.py:88-123) — a large wrong-sign SSI effect certifies.

Why this matters now

#377 showed the cost of evidence gaps in the other direction (a correct build failing); this is the same architecture failing open: a build run with --allow-input-mass-drift --allow-ecps-parity-gaps --skip-reform-coverage-smoke --allow-input-coverage-gaps publishes with a gates mapping that looks exactly like a clean run's, minus entries nobody is required to count.

Fix sketch

  1. One exhaustive gate report, written by the builder, listing every declared gate with passed | failed | skipped | waived, its criticality (release-blocking or advisory), its configuration (tolerances, allow-flags), and the SHA of its evidence file. Include it in REQUIRED_RELEASE_FILES and the release-manifest artifact list.
  2. Publication rejects a release whose gate report has any release-blocking gate missing, failed, skipped, or waived. (Waivers become impossible to ship silently — a deliberately waived gate requires a reviewed exclusion in the report itself, same #286 semantics.)
  3. Call the take-up contract assertions as a build preflight (they are cheap engine-metadata checks).
  4. Probes gain expected_direction (and optionally a plausible-range band) so wrong-sign effects fail: extends #368's Deliverable 1, does not replace it.

Acceptance criteria

  • build_manifest (or a dedicated gate report) enumerates every declared gate with verdict + enforcement status; waived/skipped gates appear as such rather than disappearing.
  • validate_release_dir fails a release with any release-blocking gate missing/failed/skipped/waived.
  • input_coverage.json, input_mass_parity.json, and reform_coverage_smoke.json are manifest-listed artifacts with SHAs.
  • assert_take_up_contract_current runs in main() before source stages.
  • A wrong-sign reform probe fails the smoke gate.

Cross-links

  • #368 / #369 (coverage gate is hard in-build; this binds it into certification)
  • #377 (cross-register consistency — the same evidence architecture, failing closed)
  • #305 (validation portfolio META), #366 (consumer-facing release evidence)

Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _build_manifests and the gate setup in tools/build_us_fiscal_refresh_release.py, then read validate_release_dir and REQUIRED_RELEASE_FILES in packages/populace-data/src/populace/data/contract.py. Review the take-up contract and reform smoke modules and their tests. Done means every declared gate and required evidence artifact is represented, and publication rejects missing, failed, skipped, or waived blocking gates, including wrong-sign probes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, release, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.