PolicyEngine / PolicyEngine/microcosm

US: pin raw PUF; move all processed-PUF derivations (aging + Section 199A simulation + carried leaves) into populace

Open
#530 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Principle

Populace should be solely based on the raw microdata, the ledger, and the rules engine — plus inflation/aging updates. Otherwise it's populace logic.

(Max, 2026-07-24, during the TAXSIM/NBER data-taxonomy discussion.)

The strict cutover removed all code dependencies on policyengine-us-data, but pinned artifacts whose contents were generated by retired code remain, and they violate the principle: derivations we can neither regenerate nor modify without resurrecting archived code.

What violates it today

The US build has exactly one inherited pin: release://policyengine/irs-soi-puf/1.8.0/puf_2024.h5 (source_stages.json, stage puf_tax_detail; asset published 2024-09-30, unchanged since). It is a processed PUF — the 2015-based file aged to 2024 by the retired pipeline — carrying content generated at archived commit policyengine-us-data@42ed5d45c56df80d754fbe24cce21cfeb8d05cbe:

  1. The 15 Section 199A leaves (puf.py L105–405, assumptions qbi_assumptions.yaml): per-source qualification flags (flat Bernoulli 60–98% by source, unconditioned), business_is_sstb (flat 30/25/15% coin flip by source — the PUF has no occupation/industry), the all-or-nothing SSTB self-employment split, W-2 wages (receipts logit calibrated to an 18% employer share + Beta labor-ratio draws), UBIA (capital-intensity Bernoulli + lognormal multiples of QBI), and qualified REIT/PTP/BDC income (Bernoulli × Beta carve-outs from dividends and partnership/S-corp income). Drawn once, seeded, frozen. populace.build.us_runtime.qbi_inputs restores the split identities after QRF placement but the draws themselves are inherited data.
  2. Retired direct-mapping leaves the manifest notes as carried by the artifact rather than re-derived: e.g. educator expense, Form 4952 elected investment income, SALT refund income ("the pinned processed 1.8.0 artifact carries nondefault source signal"), qualified tuition fallback.
  3. The 2015→2024 aging applied by the retired pipeline before export.

Consequence: improving any of these (e.g. an occupation-conditioned SSTB prior, which came up directly in today's NBER/TAXSIM discussion) currently requires resurrecting archived code to cut a new artifact. That is the opposite of the versioned-assumption auditability populace is supposed to provide.

Plan

  1. Artifact-level audit table (also an explicit #204 acceptance item that doesn't exist yet): classify every column populace consumes from puf_2024.h5 as raw-E-code passthrough vs retired-derived, with the archived-commit citation per column. The 25 42ed5d45 citations in source_stages.json are the starting index.
  2. Pin the raw PUF asset and implement PUF aging as a populace stage, under the same uprating discipline as the other sources.
  3. Port the 199A simulation into populace as a versioned stage: qbi_simulation_version=1 reproduces the archived output (exact seeded-stream reproduction if feasible; otherwise distributional-equivalence gates plus common-surface no-regression against the certified release), with the assumptions file populace-owned so a v2 (occupation-conditioned SSTB, SCF/Treasury-informed priors) becomes a normal reviewed PR.
  4. Port the remaining carried leaves the audit surfaces (item 2 above).
  5. UK: apply the same doctrine audit to the uk-data port (#144, #145, #147) so populace-uk lands raw-only from day one — no processed-artifact inheritance.

Guardrails

  • Certified releases must not shift silently: gate any pin flip with the #204 common-surface incumbent comparison.
  • Related: #416 (whether *_would_be_qualified should be derivations rather than QRF targets), #412 (reconciliation base-column contract), #298 (QBI fit vs targets). While auditing targets I also noticed the ledger row irs_soi.ty2022.historic_table_2.us.all.qbi_amount compiles to $31.3B against a modeled ~$215B (relative error 5.86 in the 2026-06-16 release diagnostics) — published TY2022 QBID is ~$200B+, so that target row looks mis-scaled at ingestion; needs verification against the current release and, if confirmed, a ledger fix.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with source_stages.json and the pinned puf_2024.h5 stage, then read the referenced puf.py sections and qbi_assumptions.yaml. Trace how populace.build.us_runtime.qbi_inputs restores QBI identities and review issue #204 for the audit-table and incumbent-comparison requirements. Done means the raw asset, aging, 199A simulation, and carried leaves have an explicit, tested populace-owned path without silent certified-release changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.