PolicyEngine / PolicyEngine/microcosm

Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns

Open
#327 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Summary

_export_input_mass_gate (in tools/build_us_fiscal_refresh_release.py) compares the calibrated export frame against the raw pre-calibration base frame with a symmetric ±50% tolerance. For a dense parent built from a raw pooled-ASEC base, this fires on exactly the PUF-imputed income columns that calibration is supposed to scale up toward SOI/CBO fiscal targets — flagging correct, target-aligned masses as failures. The gate as wired is mis-referenced for the dense-parent-from-raw-base case. It should either (a) reference a certified release instead of the raw base, or (b) thread a reviewed-exclusions register for the PUF-imputed columns, or both. It should not be silenced globally with --allow-input-mass-drift (that would also mask the genuine #278 zeroing it was built to catch).

Where it bit

Build F (populace#299), attempt 5, dense-default run on the certified 3-year ASEC pool (337,704 hh). Calibration was excellent — final_loss 0.0416, within-10% 86.1% (beating certified f0af251's 0.0423 / 86.4%) — and every structural gate passed (target_profile_coverage, degenerate_input_signal, base_population_scale, immigration_composition, ecps_parity, validation_input_coverage). The only failure was _export_input_mass_gate, which raises after the export frame exists (which is why earlier attempts, dying at the CHIP gate, never reached it).

14 columns flagged, all export >> base (gains, not losses):

column export vs base drift
long_term_capital_gains_before_response 7.02e11 / 2.12e11 +230.7%
short_term_capital_gains 9.58e10 / 2.90e10 +230.7%
miscellaneous_income 1.76e10 / -8.15e9 +315.6% (base is negative → % meaningless)
partnership_self_employment_net_earnings 3.47e10 / 1.25e10 +177.2%
qualified_dividend_income 3.20e11 / 1.32e11 +142.3%
non_sch_d_capital_gains 7.43e10 / 3.38e10 +119.7%
tax_exempt_interest_income 6.13e10 / 2.82e10 +116.9%
charitable_cash_donations 2.61e11 / 1.38e11 +88.9%
charitable_non_cash_donations 5.57e10 / 3.05e10 +82.6%
rental_income 3.84e11 / 2.13e11 +80.4%
non_qualified_dividend_income 1.35e11 / 8.22e10 +64.3%
unemployment_compensation 3.35e10 / 2.05e10 +63.3%
home_mortgage_interest 5.31e11 / 3.45e11 +54.1%
first_home_mortgage_interest 5.05e11 / 3.33e11 +51.9%

(tolerance 0.5, minimum_reference_total $1B, 41 columns checked, reviewed_exclusions empty.)

Why these masses are RIGHT (not a calibration bug)

Compared each flagged column's calibrated national mass to its SOI/CBO national target (the honest reference, vs the un-aligned raw base):

  • qualified_dividend_income: export $320B; CBO qualified-dividend target $354B → est is hitting the target (−9.7%). The gate flags +142% vs raw base for a mass that is slightly under its fiscal target.
  • capital-gains family (LT/ST/non-Sch-D): CBO net_capital_gain target $1.291T, calibrated est $0.798T → −38.2%, i.e. even after +230% vs raw base, capital gains still undershoot CBO. The increase the gate flags is genuinely needed and still insufficient.
  • partnership_self_employment_net_earnings → CBO net_business_income target $1.916T, est $1.917T → +0.1%, essentially exact.
  • AGI control: −0.9%.

CPS structurally under-reports capital, dividend, and business income; PUF imputation seeds it low; calibration reweights toward SOI/CBO. The ±50%-vs-raw-base band asks calibration not to do its job. miscellaneous_income additionally has a negative base total, making a relative-drift check ill-defined.

Why the raw-base reference is the wrong yardstick here

The gate's docstring states its purpose: "L0 selection and reweighting optimize the target surface; an untargeted input column can lose its mass without moving any residual. The export must keep every material input base." It is the #278 mass-LOSS guard (sparse selection zeroing IRA/HSA/pension/childcare — candidate << reference). It applies abs(drift) > tol symmetrically, so it also fires on calibration gains — the opposite of the defect it targets.

Provenance corroborates: input_mass_parity_gate was added in #279 (44868be), after the certified f0af251 (Jun 20) and the live-default lineage. Neither certified release manifest records input_mass_parity (only the base_population_scale person-population mass_repair, which build-f also passes). The certified 57k live default was built without this gate and would itself fail export-vs-raw-base on the same PUF columns. There is no pre-calibration income-mass alignment step in the certified pipeline.

Proposed fix

  1. Reference a certified release, not the raw base, for the export-vs-reference comparison (the underlying input_mass_parity_gate already supports reference_totals from a reference H5; the release builder already has an --input-mass-reference-h5 path used by _input_mass_reference_gate). Against a certified reference, calibration-driven upward alignment of PUF income is in-band, while a genuine sparse zeroing still fails.
  2. And/or thread reviewed_exclusions through _export_input_mass_gate (the underlying fn takes it; the wrapper currently doesn't) for the PUF-imputed columns, each with the SOI/CBO-target justification above, so the register documents why these columns legitimately move.
  3. Keep the loss/zeroing arm strict (candidate == 0 or candidate << reference must still fail — that is the #278 signature). Consider making the gain-side tolerance reference the fiscal target rather than the base, or asymmetric.
  4. Do not rely on --allow-input-mass-drift as the standing answer — it disables the whole gate, re-opening #278.

Context

Found during the Build F certified-lineage rerun (populace#299). This is the last gate the dense parent trips; with it correctly referenced, the dense build passes on the certified pool.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tools/build_us_fiscal_refresh_release.py at _export_input_mass_gate, then trace the underlying input_mass_parity_gate and the existing --input-mass-reference-h5 path used by _input_mass_reference_gate. Re-run the dense Build F certified-lineage case; done means legitimate PUF-imputed gains no longer fail while the #278-style mass-loss or zeroing case remains guarded.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.