Triage 16 reference-suspect cases from the Claude 5 failure audit

The Codex failure-audit of the Claude 5 models' 322 wrong cases (PR #90) flags 16 cases where the judge suspects the PolicyEngine reference value rather than the models — clustering in CHIP/Medicaid income-threshold handling, three state income tax cells, two SNAP cells, and Head Start eligibility. Each needs human triage against policyengine-us; the June audit's equivalents produced real upstream fixes.

The verdicts are conservative by prompt design (default is models-wrong), so these are worth taking seriously. Reproduction: every case is a frozen-manifest cell — scenario inputs in paper/snapshot/20260501/us_scenarios.csv, reference values in us_reference_outputs.csv (policyengine.py 4.16.1 / policyengine-us 1.723.0).

scenario_032 · child1_chip_eligible

PolicyEngine's CHIP reference appears to use an implausibly low Minnesota child CHIP income threshold, treating 275% FPL for a family of three as about $38,000 when it should be far higher, making the stated $42,664 income likely below rather than above the threshold.

scenario_026 · child1_medicaid_eligible

The recorded reference value appears inverted or mismatched: it says child1_medicaid_eligible = Yes while the supplied PolicyEngine derivation explicitly says PolicyEngine determined Child 1 is not Medicaid eligible (False).

scenario_064 · child1_medicaid_eligible

The recorded reference value is No, but the provided PolicyEngine derivation explicitly says PolicyEngine determined child1_medicaid_eligible is True, suggesting a reference extraction or label mismatch for this variable.

scenario_038 · child2_chip_eligible

PolicyEngine appears to apply an implausibly low Louisiana CHIP income threshold, treating 200% FPL for a four-person household as about $21,600 when 200% FPL should be far higher than this household's roughly $22,992 income.

scenario_100 · child2_head_start_eligible

PolicyEngine appears to be denying Head Start eligibility despite a 5-year-old child in a three-person household with $5,915 annual wages, which is plainly below the federal poverty guideline threshold described in the reference rationale.

scenario_008 · child3_chip_eligible

The derivation appears to apply the CHIP income threshold backwards or uses an incorrect New Jersey 2026 threshold, since an 8-person household with about $30,915 income is far below any plausible child CHIP/NJ FamilyCare income limit expressed as a multiple of FPL.

scenario_109 · child3_chip_eligible

PolicyEngine appears to be using an implausibly low 200% FPL threshold for a five-person household in 2026, causing Florida CHIP eligibility to be denied despite $30,000 being below the likely applicable income limit.

scenario_026 · child3_medicaid_eligible

PolicyEngine appears to be returning Medicaid eligibility for a 9-year-old NC child despite household income exceeding the stated child Medicaid/CHIP income threshold, possibly conflating CHIP eligibility with Medicaid eligibility or applying the wrong income limit.

scenario_008 · child5_head_start_eligible

PolicyEngine appears to compare the household's roughly $30,915 income against an incorrect Head Start poverty threshold for an 8-person household, since that income is well below the federal poverty guideline rather than above it.

scenario_015 · head_medicaid_eligible

PolicyEngine appears to have evaluated the head under Indiana's non-disabled adult Medicaid pathway despite the prompt setting is disabled=true, potentially missing a disability or working-disabled Medicaid category.

scenario_031 · head_medicaid_eligible

PolicyEngine appears to have applied California's MAGI Medicaid expansion logic for non-elderly adults despite the head being age 67, where aged Medicaid rules should govern.

scenario_038 · snap

The stated PolicyEngine derivation says the monthly SNAP allotment is $994 minus a $392.70 expected contribution, or $601.30, but $601.30 times 12 is $7,215.60 rather than the reported $7,286.94.

scenario_121 · snap

PolicyEngine appears to be applying the SNAP gross income limit to a household with a disabled member, even though elderly/disabled SNAP households are generally subject to the net income test rather than the gross income test, and the stated medical expenses could reduce net income enough to produce a positive benefit.

scenario_046 · state_income_tax_before_refundable_credits

PolicyEngine's derivation says Oklahoma itemized deductions of $22,429.78 exceeded the $12,700 standard deduction, but the taxable income of $75,303 appears to subtract the standard deduction rather than the larger itemized deduction.

scenario_056 · state_income_tax_before_refundable_credits

PolicyEngine appears to apply New Jersey's rate schedule to taxable income after the $1,000 exemption while omitting New Jersey's low-income filing/taxability threshold that eliminates liability for a single filer with gross income under $10,000.

scenario_085 · state_income_tax_before_refundable_credits

PolicyEngine appears to have incorrectly zeroed Pennsylvania income tax before refundable credits, likely by applying an erroneous income threshold/tax-forgiveness rule or excluding taxable interest income that Pennsylvania taxes at its flat rate.

🤖 Generated with Claude Code

Start with the 16 frozen-manifest cases in paper/snapshot/20260501/us_scenarios.csv and compare their values in us_reference_outputs.csv with the stated PolicyEngine 4.16.1 and policyengine-us 1.723.0 derivations. Verify each case against policyengine-us, especially the CHIP, Medicaid, Head Start, SNAP, and state-tax rules. Done means each suspect case has a confirmed reference error, an upstream issue or fix identified, or a documented reason the reference is correct.