PolicyEngine / PolicyEngine/microcosm

base-m collapses partnership_self_employment_net_earnings to signed near-cancellation (+47.7B/−48.1B legs)

Open
#432 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Found by the export-mass parity gate on Build M sparse attempt 10 (release id populace-us-2024-buildm-sparse-rmloss100-5482d38-20260715T114657Z): partnership_self_employment_net_earnings exported −$22.72B against the reference's +$61.74B (−136.8%, far beyond the ±50% band).

The gate is right that mass was lost, and the loss is in the base pool, not the solve:

artifact weighted national mass
live-default reference c2065b64 (us-data era, PUF-direct values) +$61.74B
base-j pool (865,046 persons, base weights) +$12.21B
base-m pool (same persons, rebuilt stages) −$0.35B
Build M sparse export (57,240 hh, solved weights) −$22.72B

base-m's column decomposes to a positive leg of +$47.71B against a negative leg of −$48.06B — near-perfect cancellation. The column is signed, sparse (4,467 nonzero of 865k persons, ~0.5%), and heavy-tailed on both sides: the worst case for leaf-averaged QRF imputation. base-j had already compressed it (+$61.7B → +$12.2B); the base-m rebuild (staged builder #406 chains, #415 leaf bound in effect) pushed the net to zero. The solve then amplifies in either direction because nothing pins the column — Build J's solve happened to drag it up in-band (−37.6%), Build M's dragged it down and out.

Reachability is NOT the issue (unlike the miscellaneous_income / #393 case): at ratio-5 the positive leg alone could reach $238B. The pool's signed near-cancellation is.

Remedy, in order:

  1. Diagnose the QRF chain for this column in the staged base builder: does the near-discrete guard, the leaf bound, or donor-pool construction destroy the positive/negative separation? Compare per-leg distributions donor-PUF vs imputed.
  2. Fix and rebuild base (per-target checkpointing makes this a single-chain rebuild, not a full base rebuild).
  3. Longer term (architecture week): identify the column with a real target the way Build H did for estate_income — SE-earnings concepts can anchor to SOI Schedule SE statistics — so the solve stops treating it as a free dimension. Same applies to every unpinned column the parity gate watches.

A reviewed parity exclusion for this column ships with Build M (register entry cites this issue); the exclusion lifts with the rebuild, exactly like the miscellaneous_income/#393 pattern.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the staged base builder and its QRF chain for partnership_self_employment_net_earnings, using the export-mass parity gate and the base-j/base-m comparison as context. Compare donor-PUF and imputed positive/negative leg distributions, then use per-target checkpointing to rebuild the affected chain. Done means the signed base mass is no longer near cancellation and the reviewed parity exclusion can be lifted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.