PolicyEngine / PolicyEngine/microcosm

SNAP state take-up: fill rates use pre-mass-repair weights and the export drops the eligibility surface — calibrated caseloads don't reproduce from the artifact

Open
#419 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Follow-up to #372/#376, measured on the first build to calibrate the FNS caseload targets: Build K (populace-us-2024-buildk-snapval-1fe5fc4-20260713T171826Z, staging-only, all gates green, registry b9286d4dc8a3 + 16 CHIP zero-support exclusions).

The wedge

Three measurements of the same 52 state household-caseload targets:

view result
calibrated column (release diagnostics) 52/52 within 10%
benefit-dollar targets 9 → 5 misses vs Build J, all shrunk
snap > 0 simulated from the exported h5 (what any downstream PE-US user computes) 23/52 outside 10% (Build J: 38/52)

The improvement is real (national −1.0% vs −4.7%; CA −35%→−16%; IL −31%→−14%) but the calibrated view materially overstates what the artifact delivers: AK +563%, DE +102%, KS +65%, DC +55% on simulated households.

Mechanism (verified)

The engine's take-up gating is NOT the problem: zero benefit-receiving spm_units carry takes_up_snap_if_eligible = False, and the recalibrated flags ship in the export (87.5% True vs Build J's uniform 82.1%). Two things drive the wedge:

  1. Fill rates are derived on pre-mass-repair design weights. us_snap_state_take_up.json shows AK with eligible_weight 28,689 vs target 31,319 → saturated: true, 100% flagged. But the release then mass-repairs household weights ×5.33 (Census person-population anchor) before calibration, so the real eligible pool is ~150k+ and the flag constrains nothing. A dozen states saturate this way at design weights.
  2. Build-frame is_snap_eligible ≪ export-sim eligibility. With identical shipped weights, the materialized caseload column sums to 31,264 in AK while export-simulated snap > 0 sums to 207,693 — so most sim-eligible taker units carried a 0 in the build-time column. The export's input surface (b335241 coverage vintage) omits eligibility-narrowing inputs the build frame carried; downstream simulations therefore see a much broader eligible set than the one the calibration pinned.

Proposed fixes

  • Run the take-up stage on post-repair weights (or apply the mass repair before the stage) so saturation is decided against real-world-scale pools.
  • Close the eligibility-surface gap: either export every input is_snap_eligible consumes on the frame, or persist the build-time eligibility/caseload indicator itself as a dataset column so the calibrated measure is reproducible from the artifact.
  • Add an artifact-level regression check: simulated snap > 0 per state vs the FNS table, tolerance-gated, so calibrated-vs-delivered drift fails closed instead of being discovered by audit. (Before/after scorecards from Build J/K available as the seed for this.)

Directly relevant to the SNAP work-requirements deliverable: reform runs simulate from the export, so they inherit view 3, not view 1.

Refs #370, #372, #376; Build K context in #410/#411/#412 and PR #414.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with us_snap_state_take_up.json and the exported h5 from Build K, comparing the calibrated caseload column with downstream snap > 0 totals by state. Review the post-repair weighting and export eligibility surface described in the issue, then use the Build J/K scorecards as a baseline. Done means the delivered artifact reproduces the FNS targets within the stated tolerance and includes an artifact-level regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.