PolicyEngine / PolicyEngine/microcosm-benchmarks
Certified populace-us drifts behind current eCPS on 10 parity layers; pin the parity reference + restore the gate runner
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 0
- Forks
- 0
- Avg merge
- 29m
- Merged PRs (30d)
- 1
Description
Summary
The certified US default (populace-us-2024-5da5a95-20260611, the dataset shipped in policyengine.py bundle 4.16.2) passes its build-time parity gate but drifts behind the current enhanced CPS on 10 populated layers. This was reproduced, not inferred — by pulling the build's own check_parity.py out of git history (it was removed from HEAD, see below) and running it against the sha-verified certified artifact with two references:
| Reference eCPS | parity result |
|---|---|
build's frozen reference (~/populace-score-work/enhanced_cps_2024_hf_main.h5, 237 stored layers) |
PASS, gaps=0 (169 populated layers checked) |
current eCPS (policyengine-us-data/storage/enhanced_cps_2024.h5, 215 stored layers) |
FAIL, gaps=12 (164 checked) |
Same candidate both runs (populace_us_2024.h5, sha256 f32c2e5e…, == the certified_data_artifact in policyengine.py us.json). So this is not a gate escape and not a build bug — the build's parity_gaps=0 reproduces exactly against the eCPS it was gated on. The reference moved; the parity gate ran against an unpinned, since-changed eCPS, and nothing re-checks for drift.
exported_nonzero passes (308 stored columns, all non-zero) — populace's own columns are clean. The gaps are purely missing relative to the newer eCPS.
The 12 gaps vs current eCPS
Pure inputs (no formula) — add to the impute/donor set (PUF-derived):
amt_foreign_tax_credit(eCPS populates 9.3% of records)general_business_credit(5.4%)excess_withheld_payroll_tax(4.6%)prior_year_minimum_tax_credit(4.0%)early_withdrawal_penalty(3.3%)other_credits(2.8%)
These feed net federal tax, so the certified default computes different (lower-magnitude) tax than current eCPS for the affected 3–9% of records.
Formula-owned but yields 0 — populate the upstream inputs the formula needs:
self_employed_health_insurance_ald(7.4%) — registered formula exists, but its upstream self-employed health-premium inputs are absent in populace, so it computes 0.
Reported / comparison aggregates (not formula-consumed) — decide carry-or-document:
spm_unit_net_income_reported(99.4%),spm_unit_total_income_reported(97.7%),ssi_reported(1.5%)
Vacuous (eCPS populates 0% too — no action):
recapture_of_investment_credit,unreported_payroll_tax
Root-cause / process gaps (the more important fixes)
- The parity reference is unpinned. The gate ran against a working-copy eCPS, not a recorded revision, so "parity 0" is not reproducible and silently rots when eCPS changes.
- The certified release manifest records no gate result — no gaps count, no reference identity, no skipped-layer count. The verdict isn't in the artifact.
- The gate runner was deleted from HEAD.
packages/populace-data/build/us/check_parity.py(and the rest of thebuild/ussnapshot) were removed infda3838 "Remove retired build and benchmark surfaces". The gate library (populace.build.parity_gate) remains, but nothing invokes it against a reference — so the release contract is only reconstructable from git history.
Proposed fixes
- Re-home a reference-pinned simulation-level parity runner into
populace-build(reuseparity_gate), recording the reference eCPS repo+revision+sha256 and the full result (gaps, skipped, layer counts) in the release manifest. (PR incoming.) - CI drift-check: re-run parity against the latest published eCPS on every build so reference drift fails loudly instead of going invisible.
- Close the 6 pure-input credit/penalty gaps via the PUF donor/impute stage; populate the upstream for
self_employed_health_insurance_ald; decide explicitly whether to carry the 3 reported aggregates or document them as a named exemption. - Restore the build/gate surface to the repo (or move it into the package) so the release contract is auditable, not archived in a deleted commit.
Repro: check_parity.py from commit 33ed83d, run with POP_CANDIDATE/POP_REFERENCE env overrides.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the historical packages/populace-data/build/us/check_parity.py at commit 33ed83d and the remaining populace.build.parity_gate library, then inspect policyengine.py's us.json certified artifact entry. Reproduce the parity check with the recorded candidate and references. Done should include an auditable pinned reference, recorded parity results, and a restored or relocated gate runner with drift checking addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, ci-cd, data-engineering, release
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100