PolicyEngine / PolicyEngine/microcosm
PUF donor uprating overstates mean-reverting distribution series: non_sch_d at 2.39x the SOI concept
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Found adjudicating Build N attempt 2 (#462 thread). Measured on the sha-pinned puf_2024.h5 donor via puf_tax_unit_donor_from_arrays:
| donor (E01100 lineage, uprated to 2024) | SOI Pub 1304 Table 1.4 TY2023 | |
|---|---|---|
| weighted carriers | 4.67M | 3.21M (1.45×) |
| weighted mass | $24.31B | $10.16B (2.39×) |
| weighted mean | $5,206 | $3,165 (1.64×) |
| weighted median | $1,450 | — |
The eCPS-era pipeline produced $13.69B / 4.01M units from the same PUF lineage (recorded in #462's motivation), so the current ETL's 2024-level uprating roughly doubles what the retired pipeline carried. Capital-gain distributions are a mean-reverting series (TY2015 — the PUF base year — was a historically heavy distribution year); uprating its level by a generic growth factor overstates the modern concept on both margins.
Interim: #486 pins the column's level to the ledger-fed aged Table 1.4 dollar fact at release time (the SS-components repair precedent), with the factor recorded in support_value_repairs. That is honest but uniform — it preserves the donor's (too-rich) shape.
Root fix: variable-specific donor uprating for distribution-class series — age E01100-class amounts by their own SOI chain (the #474 tips mechanism generalized) either in the irs-soi-puf ETL or at populace donor load, with the chain facts in the ledger. Candidates beyond non_sch_d: the other Table 1.4 rows the #464 blanket watches, and the 13 tail-concentrated sparse columns from the N attempt-2 battery (those also implicate #481's weighted-leaf-draw loss).
Relates #462, #464, #481, #486.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing puf_tax_unit_donor_from_arrays and the current irs-soi-puf ETL or populace donor load path, then compare the donor figures with the ledger facts described in #486 and the findings in #462. The fix is complete when distribution-class series use variable-specific SOI uprating with chain facts recorded in the ledger, including coverage for the related Table 1.4 and sparse-column cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100