PolicyEngine / PolicyEngine/microcosm-benchmarks

Certified populace-us drifts behind current eCPS on 10 parity layers; pin the parity reference + restore the gate runner

Open
#1 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
0
Forks
0
Avg merge
29m
Merged PRs (30d)
1

Description

Summary

The certified US default (populace-us-2024-5da5a95-20260611, the dataset shipped in policyengine.py bundle 4.16.2) passes its build-time parity gate but drifts behind the current enhanced CPS on 10 populated layers. This was reproduced, not inferred — by pulling the build's own check_parity.py out of git history (it was removed from HEAD, see below) and running it against the sha-verified certified artifact with two references:

Reference eCPS parity result
build's frozen reference (~/populace-score-work/enhanced_cps_2024_hf_main.h5, 237 stored layers) PASS, gaps=0 (169 populated layers checked)
current eCPS (policyengine-us-data/storage/enhanced_cps_2024.h5, 215 stored layers) FAIL, gaps=12 (164 checked)

Same candidate both runs (populace_us_2024.h5, sha256 f32c2e5e…, == the certified_data_artifact in policyengine.py us.json). So this is not a gate escape and not a build bug — the build's parity_gaps=0 reproduces exactly against the eCPS it was gated on. The reference moved; the parity gate ran against an unpinned, since-changed eCPS, and nothing re-checks for drift.

exported_nonzero passes (308 stored columns, all non-zero) — populace's own columns are clean. The gaps are purely missing relative to the newer eCPS.

The 12 gaps vs current eCPS

Pure inputs (no formula) — add to the impute/donor set (PUF-derived):

  • amt_foreign_tax_credit (eCPS populates 9.3% of records)
  • general_business_credit (5.4%)
  • excess_withheld_payroll_tax (4.6%)
  • prior_year_minimum_tax_credit (4.0%)
  • early_withdrawal_penalty (3.3%)
  • other_credits (2.8%)

These feed net federal tax, so the certified default computes different (lower-magnitude) tax than current eCPS for the affected 3–9% of records.

Formula-owned but yields 0 — populate the upstream inputs the formula needs:

  • self_employed_health_insurance_ald (7.4%) — registered formula exists, but its upstream self-employed health-premium inputs are absent in populace, so it computes 0.

Reported / comparison aggregates (not formula-consumed) — decide carry-or-document:

  • spm_unit_net_income_reported (99.4%), spm_unit_total_income_reported (97.7%), ssi_reported (1.5%)

Vacuous (eCPS populates 0% too — no action):

  • recapture_of_investment_credit, unreported_payroll_tax

Root-cause / process gaps (the more important fixes)

  1. The parity reference is unpinned. The gate ran against a working-copy eCPS, not a recorded revision, so "parity 0" is not reproducible and silently rots when eCPS changes.
  2. The certified release manifest records no gate result — no gaps count, no reference identity, no skipped-layer count. The verdict isn't in the artifact.
  3. The gate runner was deleted from HEAD. packages/populace-data/build/us/check_parity.py (and the rest of the build/us snapshot) were removed in fda3838 "Remove retired build and benchmark surfaces". The gate library (populace.build.parity_gate) remains, but nothing invokes it against a reference — so the release contract is only reconstructable from git history.

Proposed fixes

  • Re-home a reference-pinned simulation-level parity runner into populace-build (reuse parity_gate), recording the reference eCPS repo+revision+sha256 and the full result (gaps, skipped, layer counts) in the release manifest. (PR incoming.)
  • CI drift-check: re-run parity against the latest published eCPS on every build so reference drift fails loudly instead of going invisible.
  • Close the 6 pure-input credit/penalty gaps via the PUF donor/impute stage; populate the upstream for self_employed_health_insurance_ald; decide explicitly whether to carry the 3 reported aggregates or document them as a named exemption.
  • Restore the build/gate surface to the repo (or move it into the package) so the release contract is auditable, not archived in a deleted commit.

Repro: check_parity.py from commit 33ed83d, run with POP_CANDIDATE/POP_REFERENCE env overrides.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the historical packages/populace-data/build/us/check_parity.py at commit 33ed83d and the remaining populace.build.parity_gate library, then inspect policyengine.py's us.json certified artifact entry. Reproduce the parity check with the recorded candidate and references. Done should include an auditable pinned reference, recorded parity results, and a restored or relocated gate runner with drift checking addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd, data-engineering, release
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.