PolicyEngine / PolicyEngine/microcosm
Loss shape: flat cap at 1.0 makes past-cap rows free dumping grounds (17 rows pushed out in Build N); bounded-tail experiment + past-cap census diagnostic
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Problem: the flat loss cap creates free dumping grounds
The calibration objective is capped weighted MAPE: weighted_mean(min(|rel_err|, cap)) with US_FISCAL_TARGET_LOSS_CAP = 1.0. Past the cap, a row's gradient is exactly zero — the 101st percentage point of error is free.
Census of the published Build N sparse run (populace-us-2024-buildn-sparse-rmloss100-c3e378a-20260722T010408Z), 5,672 targets:
| count | |
|---|---|
| past cap at initialization | 929 |
| escaped back inside during the solve | 569 |
| past cap at final | 377 |
| frozen (past at init and final) | 360 |
| pushed out (in-cap at init → past-cap at final) | 17 |
Two live harms, both measured:
- The ratchet/dumping dynamic. 17 rows that started alive were pushed past the cap and abandoned this run — LA TANF basic assistance went 0.47 → 5.54, and a cluster of small state-AGI taxable-interest bins went 0.1–0.8 → 2.0–3.5 (the #488-aged interest family squeezed, and the overflow poured into rows with zero marginal cost). Four of the six broken medical states got worse while past cap (DC 1.34→3.78, UT 1.42→3.93, TN 1.92→2.09, NV 1.60→1.80): their fat carriers were upweighted for other targets and the medical side effect was free. See the #462 medical decomposition comment — the national +20.78% medical miss is exactly the six-state pinned excess.
- Silent triage. Nothing reports which rows the solve wrote off. The 377 past-cap rows and especially the 17 pushed-out rows should be first-class release diagnostics.
What the cap gets RIGHT (do not lose this): 929 rows start past it — including near-zero-denominator cells at relative errors of 10⁸ (dense EITC) and 347×-off state rental rows. Uncapped, those own the whole gradient. And 61% of initially-dead rows recover anyway via shared carriers. Bounded influence must be preserved.
Proposal
A. Past-cap census as a standard release diagnostic (independent of any loss change): init-past / final-past / escaped / pushed-out counts + the pushed-out row list, in calibration diagnostics and the scorecard. Cheap, pure observability.
B. Loss-shape experiment: replace the flat tail with a bounded, nonzero-gradient tail. Candidates, both uniform and global (no per-target anything):
- Leaky cap (minimal diff):
min(r, cap) + ε·max(r − cap, 0), ε ≈ 0.02. Bitwise identical to today below the knee; constant small tail gradient kills the ratchet. - Sigmoid family (smooth): a scaled rational such as
(1+c)·r/(c+r). Constraints that pick the shape: slope ≈ 1 at r=0 (keep the sharp median-seeking pull where ~89% of targets live — rules out Cauchy/Geman-McClure, which are quadratic near zero), and do NOT soften the 0.1–1.0 shoulder (raw tanh/logistic drops shoulder gradient to ~0.5 at r=0.9 — taxing exactly the rows still being fought for; c ≈ 3 keeps shoulder ≈ 0.75 at the knee with a polynomial tail ≈ 0.24 at r=4).
Protocol: rerun the attempt-6 configuration under each candidate; score on the SAME fixed yardstick (uncapped evaluation metrics, not each run's own training loss): pushed-out count, six-state medical rows, within-10%, final loss, the 14-column tail-concentration register, ESS. Hypothesis: pushed-out → ~0 and the dumping stops; AR-class support defects remain (owners #481/#487 — a loss shape cannot fix support).
Non-goals: no per-target loss knobs (the #491 critical-row multiplier is being removed from that PR per the same principle); no cap-level raise (moves the cliff, reweights the objective against the within-10% band).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attempt-6 calibration configuration and the calibration diagnostics and scorecard entry points. Compare the leaky-cap and sigmoid candidates using the same fixed uncapped evaluation yardstick, including pushed-out counts, medical rows, within-10% results, tail concentration, and ESS. Done means the census is reported and the experiment results are recorded without adding per-target knobs or changing the cap level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- analytics, data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100