PolicyEngine / PolicyEngine/microcosm-benchmarks

Is the populace-us ↔ current-eCPS parity drift blocking downstream (CRFB taxation-of-benefits, PolicyBench)?

Open
#2 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
0
Forks
0
Avg merge
29m
Merged PRs (30d)
1

Description

Question

Is the parity drift between the certified populace-us default and the current enhanced CPS blocking for downstream consumers — specifically the CRFB taxation-of-benefits work and PolicyBench? This issue exists to collect that impact assessment. The fix for the drift mechanism is tracked separately in PolicyEngine/populace-benchmarks#1; this issue is only about whether the current state blocks downstream use while that lands.

The facts (reproduced, not inferred)

Running the build's own parity checker against the sha-verified certified artifact (populace_us_2024.h5, sha256 f32c2e5e…, == the certified default in policyengine.py bundle 4.16.2):

  • vs the eCPS the build was gated against → 0 gaps (the recorded "parity 0" is real and reproduces)
  • vs today's eCPS → 12 gaps

It is reference drift, not a gate escape: eCPS has since populated layers the certified populace zeroes, and the parity reference was never pinned. exported_nonzero still passes (populace's own columns are clean).

What diverges (current eCPS populates, certified populace = 0)

Federal tax inputs that feed net tax (no formula recovery — genuinely 0 in populace):

variable share of records eCPS populates
amt_foreign_tax_credit 9.3%
general_business_credit 5.4%
excess_withheld_payroll_tax 4.6%
prior_year_minimum_tax_credit 4.0%
early_withdrawal_penalty 3.3%
other_credits 2.8%
self_employed_health_insurance_ald 7.4% (formula-owned; 0 because upstream SE-health inputs absent)

Reported / comparison aggregates (not formula-consumed; used in reported-vs-modeled analysis):
spm_unit_net_income_reported (99.4%), spm_unit_total_income_reported (97.7%), ssi_reported (1.5%).

Not affected: retirement contributions (roth_ira/roth_401k/traditional_401k/SE-pension) are exported as _desired inputs and recovered by the engine formula (~15–17% nonzero in simulation) — not gaps.

Likely impact surface

For the ~3–9% of records each credit touches, populace computes lower-magnitude federal tax than current eCPS (those credits/penalties are simply absent). Distributional and revenue figures for affected slices will differ from an eCPS-based run. The reported aggregates only matter to analyses that read the reported (not modeled) columns.

For the CRFB taxation-of-benefits agent

The TOB v2 work runs on the populace base. Question: do any of the above — the six federal credits/penalties, the SE-health ALD, or the spm_unit_*_reported aggregates — enter the taxation-of-benefits analysis path or its targets/validation? If TOB is driven by Social Security taxability, AGI thresholds, and the standard income variables (which are unaffected), this may be immaterial; if it nets against these credits or compares to reported SPM income, it may not be. Please assess whether this blocks the TOB numbers or is tolerable until PolicyEngine/populace-benchmarks#1 closes the data gaps.

For the PolicyBench agent

Does PolicyBench score anything that depends on these layers — i.e., do any benchmark cases exercise the six credits/penalties or compare against eCPS on the reported aggregates? If the benchmark suite would shift (or if it pins eCPS as a reference that populace is now expected to match), quantify the score delta and state whether it blocks.

Status of the fix (context, not the question here)

PolicyEngine/populace-benchmarks#1 is closing the mechanism: a reference-pinned, recorded, reconstructable parity runner (branch pin-parity-reference, in review) so "parity 0" is reproducible against an explicit eCPS revision, plus a CI drift-check. The data gaps (importing the six PUF-derived credits; populating SE-health upstream) are the open work that this blocking assessment should prioritize or de-prioritize.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the build's parity checker against the certified populace_us_2024.h5 artifact and reviewing the certified default in policyengine.py bundle 4.16.2. Then inspect the CRFB taxation-of-benefits path and PolicyBench cases for dependencies on the listed credits, SE-health ALD, or reported aggregates. Done means documenting whether each path is affected, quantifying any benchmark shift, and stating whether downstream work is blocked.

Written by the indexing model from the issue text.

Assessment

Domain
analytics, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.