PolicyEngine / PolicyEngine/policybench
Align benchmark prompts with PolicyEngine target-year input values
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 12h 7m
- Merged PRs (30d)
- 13
Description
PolicyBench error audit found prompt/reference mismatches that can make model errors look larger or qualitatively different than they are.\n\nIssues to fix before the next paid run:\n\n- UK prompts currently show raw transfer-dataset input amounts, while the 2026-27 PolicyEngine reference simulation uprates inputs before calculating outputs. In the 100-household run, all 69 UK scenarios with positive employment income differed between prompt and PE calculation, with median PE/prompt ratio about 1.034. Capital gains, savings interest, dividends, and private pensions were also uprated before reference calculation.\n- UK pension contribution wording is ambiguous. Models interpreted employee pension contributions as reducing taxable pay, while the PE reference did not always reduce taxable income that way. Prompts should either expose target-year tax variables in the same semantics PE uses or specify the pension contribution treatment precisely.\n- ACA PTC prompts are not clean no-web household calculations when local SLCSP/benchmark premium and Marketplace enrollment assumptions are implicit. Either supply the benchmark/selected plan facts, label PTC as a no-web local-market stress test, or exclude/stratify those cases.\n- SSDI/disability plus Medicare inference is a prompt-contract risk: some models inferred Medicare from SSDI even though unlisted statuses are false and PE had Medicare eligibility false. Clarify or avoid cases where waiting-period/enrollment facts are not supplied.\n\nSuggested acceptance criteria:\n\n- Scenario prompt values are generated from the same target-year PolicyEngine values used to produce reference outputs, or the prompt explicitly says the displayed values are base-year values and gives the uprating rule. Prefer target-year values.\n- Add a test that UK prompt-visible income/gain/pension inputs reconcile to PE target-year input variables for sampled scenarios.\n- Add prompt-contract tests or snapshot examples for PTC and SSDI/Medicare edge cases.\n\nContext: this came from the post-run error audit for results/local/full_run_20260513_policyengine_4_4_4_nested_outputs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the scenario prompt generation and reference-output inputs associated with results/local/full_run_20260513_policyengine_4_4_4_nested_outputs. Compare UK income, gain, pension, PTC, and SSDI/Medicare prompt values with the target-year PolicyEngine variables. Done means sampled UK values reconcile and prompt-contract tests or snapshot examples cover the PTC and SSDI/Medicare edge cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100