PolicyEngine / PolicyEngine/policybench

scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP

Open
#92 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

Problem

scenario_121 (SC, single 53-year-old) intends a disabled household: the prompt narrates disability, and the inputs set is_disabled: true plus disability_benefits: 13044. But neither input is USDA-qualifying disability under 7 CFR 271.2 (which is receipt-based: SSI, SSDI, VA disability, etc.):

  • disability_benefits is PolicyEngine's employer/private disability income variable — not a qualifying federal program
  • is_disabled is a bare boolean that PE reads for the SNAP work-registration exemption but not for is_usda_disabled

So the reference pipeline computes a non-disabled SNAP unit: the household fails the 130% gross-income test (ratio 2.55) and gets $0, and the $30k of medical expenses never unlocks the excess-medical deduction. Force is_usda_disabled=true and the same household is eligible with $286/mo ($3,457/yr): elderly/disabled units are exempt from the gross test, and net income collapses to ~$41 after the medical deduction.

The scenario is internally ill-posed rather than PE being wrong — the ground-truth household doesn't represent what the prompt describes. Full trace in PolicyEngine/policybench#91 (row 13).

Options for the next reference refresh

  1. Preferred: re-encode the disability as a receipt-based input PE consumes for is_usda_disabled — e.g. route the disability income to social_security_disability (SSDI receipt, SGA-independent). Note the SSI route does not work here: $25,665 wages = $2,139/mo > the 2026 SGA limit.
  2. Alternatively, drop the SNAP output for this scenario as a scope gap.

Either changes a frozen scenario/reference, so it belongs in the next snapshot/reference-refresh cycle, not a live patch. Models that treated the household as disabled-for-SNAP and answered a positive amount are currently scored wrong on this cell.

Also worth considering

A generator-side lint: any scenario whose prompt narrates disability should set at least one receipt-based disabled_programs input (is_ssi_disabled chain, social_security_disability, veteran flags), since is_disabled alone propagates inconsistently in policyengine-us (work-registration yes, gross-test exemption no).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing scenario_121 and the full trace in PolicyEngine/policybench#91, then inspect how the next snapshot/reference-refresh cycle stores scenario inputs and outputs. Confirm the receipt-based disability input and expected SNAP result against the cited USDA rule; done means the refreshed reference no longer scores the described disabled household as zero, or the SNAP output is removed if that scope gap remains.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.