PolicyEngine / PolicyEngine/microcosm

PUF donor uprating overstates mean-reverting distribution series: non_sch_d at 2.39x the SOI concept

Open
#487 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Found adjudicating Build N attempt 2 (#462 thread). Measured on the sha-pinned puf_2024.h5 donor via puf_tax_unit_donor_from_arrays:

donor (E01100 lineage, uprated to 2024) SOI Pub 1304 Table 1.4 TY2023
weighted carriers 4.67M 3.21M (1.45×)
weighted mass $24.31B $10.16B (2.39×)
weighted mean $5,206 $3,165 (1.64×)
weighted median $1,450

The eCPS-era pipeline produced $13.69B / 4.01M units from the same PUF lineage (recorded in #462's motivation), so the current ETL's 2024-level uprating roughly doubles what the retired pipeline carried. Capital-gain distributions are a mean-reverting series (TY2015 — the PUF base year — was a historically heavy distribution year); uprating its level by a generic growth factor overstates the modern concept on both margins.

Interim: #486 pins the column's level to the ledger-fed aged Table 1.4 dollar fact at release time (the SS-components repair precedent), with the factor recorded in support_value_repairs. That is honest but uniform — it preserves the donor's (too-rich) shape.

Root fix: variable-specific donor uprating for distribution-class series — age E01100-class amounts by their own SOI chain (the #474 tips mechanism generalized) either in the irs-soi-puf ETL or at populace donor load, with the chain facts in the ledger. Candidates beyond non_sch_d: the other Table 1.4 rows the #464 blanket watches, and the 13 tail-concentrated sparse columns from the N attempt-2 battery (those also implicate #481's weighted-leaf-draw loss).

Relates #462, #464, #481, #486.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing puf_tax_unit_donor_from_arrays and the current irs-soi-puf ETL or populace donor load path, then compare the donor figures with the ledger facts described in #486 and the findings in #462. The fix is complete when distribution-class series use variable-specific SOI uprating with chain facts recorded in the ledger, including coverage for the related Table 1.4 and sparse-column cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.