PolicyEngine / PolicyEngine/microcosm
SNAP state take-up: fill rates use pre-mass-repair weights and the export drops the eligibility surface — calibrated caseloads don't reproduce from the artifact
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Follow-up to #372/#376, measured on the first build to calibrate the FNS caseload targets: Build K (populace-us-2024-buildk-snapval-1fe5fc4-20260713T171826Z, staging-only, all gates green, registry b9286d4dc8a3 + 16 CHIP zero-support exclusions).
The wedge
Three measurements of the same 52 state household-caseload targets:
| view | result |
|---|---|
| calibrated column (release diagnostics) | 52/52 within 10% |
| benefit-dollar targets | 9 → 5 misses vs Build J, all shrunk |
snap > 0 simulated from the exported h5 (what any downstream PE-US user computes) |
23/52 outside 10% (Build J: 38/52) |
The improvement is real (national −1.0% vs −4.7%; CA −35%→−16%; IL −31%→−14%) but the calibrated view materially overstates what the artifact delivers: AK +563%, DE +102%, KS +65%, DC +55% on simulated households.
Mechanism (verified)
The engine's take-up gating is NOT the problem: zero benefit-receiving spm_units carry takes_up_snap_if_eligible = False, and the recalibrated flags ship in the export (87.5% True vs Build J's uniform 82.1%). Two things drive the wedge:
- Fill rates are derived on pre-mass-repair design weights.
us_snap_state_take_up.jsonshows AK witheligible_weight28,689 vs target 31,319 →saturated: true, 100% flagged. But the release then mass-repairs household weights ×5.33 (Census person-population anchor) before calibration, so the real eligible pool is ~150k+ and the flag constrains nothing. A dozen states saturate this way at design weights. - Build-frame
is_snap_eligible≪ export-sim eligibility. With identical shipped weights, the materialized caseload column sums to 31,264 in AK while export-simulatedsnap > 0sums to 207,693 — so most sim-eligible taker units carried a 0 in the build-time column. The export's input surface (b335241 coverage vintage) omits eligibility-narrowing inputs the build frame carried; downstream simulations therefore see a much broader eligible set than the one the calibration pinned.
Proposed fixes
- Run the take-up stage on post-repair weights (or apply the mass repair before the stage) so saturation is decided against real-world-scale pools.
- Close the eligibility-surface gap: either export every input
is_snap_eligibleconsumes on the frame, or persist the build-time eligibility/caseload indicator itself as a dataset column so the calibrated measure is reproducible from the artifact. - Add an artifact-level regression check: simulated
snap > 0per state vs the FNS table, tolerance-gated, so calibrated-vs-delivered drift fails closed instead of being discovered by audit. (Before/after scorecards from Build J/K available as the seed for this.)
Directly relevant to the SNAP work-requirements deliverable: reform runs simulate from the export, so they inherit view 3, not view 1.
Refs #370, #372, #376; Build K context in #410/#411/#412 and PR #414.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with us_snap_state_take_up.json and the exported h5 from Build K, comparing the calibrated caseload column with downstream snap > 0 totals by state. Review the post-repair weighting and export eligibility surface described in the issue, then use the Build J/K scorecards as a baseline. Done means the delivered artifact reproduces the FNS targets within the stated tolerance and includes an artifact-level regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100