PolicyEngine / PolicyEngine/policybench

Align benchmark prompts with PolicyEngine target-year input values

Open
#19 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

PolicyBench error audit found prompt/reference mismatches that can make model errors look larger or qualitatively different than they are.\n\nIssues to fix before the next paid run:\n\n- UK prompts currently show raw transfer-dataset input amounts, while the 2026-27 PolicyEngine reference simulation uprates inputs before calculating outputs. In the 100-household run, all 69 UK scenarios with positive employment income differed between prompt and PE calculation, with median PE/prompt ratio about 1.034. Capital gains, savings interest, dividends, and private pensions were also uprated before reference calculation.\n- UK pension contribution wording is ambiguous. Models interpreted employee pension contributions as reducing taxable pay, while the PE reference did not always reduce taxable income that way. Prompts should either expose target-year tax variables in the same semantics PE uses or specify the pension contribution treatment precisely.\n- ACA PTC prompts are not clean no-web household calculations when local SLCSP/benchmark premium and Marketplace enrollment assumptions are implicit. Either supply the benchmark/selected plan facts, label PTC as a no-web local-market stress test, or exclude/stratify those cases.\n- SSDI/disability plus Medicare inference is a prompt-contract risk: some models inferred Medicare from SSDI even though unlisted statuses are false and PE had Medicare eligibility false. Clarify or avoid cases where waiting-period/enrollment facts are not supplied.\n\nSuggested acceptance criteria:\n\n- Scenario prompt values are generated from the same target-year PolicyEngine values used to produce reference outputs, or the prompt explicitly says the displayed values are base-year values and gives the uprating rule. Prefer target-year values.\n- Add a test that UK prompt-visible income/gain/pension inputs reconcile to PE target-year input variables for sampled scenarios.\n- Add prompt-contract tests or snapshot examples for PTC and SSDI/Medicare edge cases.\n\nContext: this came from the post-run error audit for results/local/full_run_20260513_policyengine_4_4_4_nested_outputs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the scenario prompt generation and reference-output inputs associated with results/local/full_run_20260513_policyengine_4_4_4_nested_outputs. Compare UK income, gain, pension, PTC, and SSDI/Medicare prompt values with the target-year PolicyEngine variables. Done means sampled UK values reconcile and prompt-contract tests or snapshot examples cover the PTC and SSDI/Medicare edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.