PolicyEngine / PolicyEngine/microcosm
US: pin raw PUF; move all processed-PUF derivations (aging + Section 199A simulation + carried leaves) into populace
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Principle
Populace should be solely based on the raw microdata, the ledger, and the rules engine — plus inflation/aging updates. Otherwise it's populace logic.
(Max, 2026-07-24, during the TAXSIM/NBER data-taxonomy discussion.)
The strict cutover removed all code dependencies on policyengine-us-data, but pinned artifacts whose contents were generated by retired code remain, and they violate the principle: derivations we can neither regenerate nor modify without resurrecting archived code.
What violates it today
The US build has exactly one inherited pin: release://policyengine/irs-soi-puf/1.8.0/puf_2024.h5 (source_stages.json, stage puf_tax_detail; asset published 2024-09-30, unchanged since). It is a processed PUF — the 2015-based file aged to 2024 by the retired pipeline — carrying content generated at archived commit policyengine-us-data@42ed5d45c56df80d754fbe24cce21cfeb8d05cbe:
- The 15 Section 199A leaves (
puf.pyL105–405, assumptionsqbi_assumptions.yaml): per-source qualification flags (flat Bernoulli 60–98% by source, unconditioned),business_is_sstb(flat 30/25/15% coin flip by source — the PUF has no occupation/industry), the all-or-nothing SSTB self-employment split, W-2 wages (receipts logit calibrated to an 18% employer share + Beta labor-ratio draws), UBIA (capital-intensity Bernoulli + lognormal multiples of QBI), and qualified REIT/PTP/BDC income (Bernoulli × Beta carve-outs from dividends and partnership/S-corp income). Drawn once, seeded, frozen.populace.build.us_runtime.qbi_inputsrestores the split identities after QRF placement but the draws themselves are inherited data. - Retired direct-mapping leaves the manifest notes as carried by the artifact rather than re-derived: e.g. educator expense, Form 4952 elected investment income, SALT refund income ("the pinned processed 1.8.0 artifact carries nondefault source signal"), qualified tuition fallback.
- The 2015→2024 aging applied by the retired pipeline before export.
Consequence: improving any of these (e.g. an occupation-conditioned SSTB prior, which came up directly in today's NBER/TAXSIM discussion) currently requires resurrecting archived code to cut a new artifact. That is the opposite of the versioned-assumption auditability populace is supposed to provide.
Plan
- Artifact-level audit table (also an explicit #204 acceptance item that doesn't exist yet): classify every column populace consumes from
puf_2024.h5as raw-E-code passthrough vs retired-derived, with the archived-commit citation per column. The 2542ed5d45citations insource_stages.jsonare the starting index. - Pin the raw PUF asset and implement PUF aging as a populace stage, under the same uprating discipline as the other sources.
- Port the 199A simulation into populace as a versioned stage:
qbi_simulation_version=1reproduces the archived output (exact seeded-stream reproduction if feasible; otherwise distributional-equivalence gates plus common-surface no-regression against the certified release), with the assumptions file populace-owned so a v2 (occupation-conditioned SSTB, SCF/Treasury-informed priors) becomes a normal reviewed PR. - Port the remaining carried leaves the audit surfaces (item 2 above).
- UK: apply the same doctrine audit to the uk-data port (#144, #145, #147) so populace-uk lands raw-only from day one — no processed-artifact inheritance.
Guardrails
- Certified releases must not shift silently: gate any pin flip with the #204 common-surface incumbent comparison.
- Related: #416 (whether
*_would_be_qualifiedshould be derivations rather than QRF targets), #412 (reconciliation base-column contract), #298 (QBI fit vs targets). While auditing targets I also noticed the ledger rowirs_soi.ty2022.historic_table_2.us.all.qbi_amountcompiles to $31.3B against a modeled ~$215B (relative error 5.86 in the 2026-06-16 release diagnostics) — published TY2022 QBID is ~$200B+, so that target row looks mis-scaled at ingestion; needs verification against the current release and, if confirmed, a ledger fix.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with source_stages.json and the pinned puf_2024.h5 stage, then read the referenced puf.py sections and qbi_assumptions.yaml. Trace how populace.build.us_runtime.qbi_inputs restores QBI identities and review issue #204 for the audit-table and incumbent-comparison requirements. Done means the raw asset, aging, 199A simulation, and carried leaves have an explicit, tested populace-owned path without silent certified-release changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100