PolicyEngine / PolicyEngine/policyengine-uk-chat

Add reform-API eval cases to catch silent regressions like the 1pp basic-rate failure

Open
#82 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
1
Avg merge
16h 46m
Merged PRs (30d)
2

Description

Background

Today's live test (2026-05-28) exposed the agent's inability to express a basic-rate +1pp reform via the PolicyEngine UK API. The failure was silent — no test, no eval signal — and only surfaced because a human watched the stream.

PR #52 (feat/eval-harness) is the right home for systematic regression coverage like this. This issue tracks adding a concrete set of reform-API cases once #52 lands.

Cases to add

Each case = a user prompt + expected behavior:

  1. Basic rate +1pp: "What's the distributional impact of raising the income tax basic rate from 20% to 21%?" — expect: positive revenue change ~£6–8bn, top deciles bearing the burden, optional chart.
  2. Personal allowance to £15k: "What if the personal allowance were £15,000?" — expect: revenue drop, bottom-quintile gain.
  3. NI primary threshold change: "What if the NI primary threshold went up by £1,000?" — expect: low-to-mid earners benefit, revenue cost.
  4. Child benefit uprating: "Uprate child benefit by 10%." — expect: lower-decile gain, modest revenue cost.
  5. Two-band income tax: "What if we collapsed basic and higher rates to a single 25% band?" — expect: a sensible reform structure, not a guessed one.

For each: assert the direction of revenue change is correct, the magnitude is within an order of magnitude of a known answer, and the agent's response either (a) emits a chart when chartsMode=on, or (b) cites the typed tool used.

Why this matters

Without these eval cases, fixes to system-prompt recipes (#filed-companion), typed tools (#filed-companion), and model selection (#filed-companion) can regress silently. Each PR that touches reform handling should run these cases.

Depends on

PR #52 landing (eval harness scaffolding).

Constraints

  • Cases should be cheap to run — single household / minimal microsim where possible.
  • Pin a dataset version (e.g. EFRS 2025/26) so case results are reproducible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PR #52's eval harness once it lands, then run a minimal reform case using the pinned EFRS 2025/26 dataset. Add the five listed prompts and assert revenue direction, order-of-magnitude accuracy, and chart or typed-tool evidence as applicable. Done means each case is cheap, reproducible, and runs for reforms touching reform handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.