PolicyEngine / PolicyEngine/policyengine-scorecard

Take-up assumptions: compare PolicyEngine and UKMOD where they are comparable

Open
#130 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6d 12h
Merged PRs (30d)
32

Description

Found while researching UKMOD (CeMPA/Essex) for scorecard comparison. Verified by reading the take-up parameters out of a real policyengine-uk 2.89.2 — the certified engine — not from documentation.

The finding

PolicyEngine-UK's take-up parameters, read from the engine:

parameter value dated from
gov/dwp/housing_benefit/takeup 1.0 2019-01-01
gov/dwp/income_support/takeup 1.0 2019-01-01
gov/dwp/tax_credits/child_tax_credit/takeup 1.0 2019-01-01
gov/dwp/tax_credits/working_tax_credit/takeup 1.0 2019-01-01
gov/dwp/universal_credit/takeup_rate 0.55 2015-01-01
gov/dwp/pension_credit/takeup 0.70 2015-01-01
gov/dwp/JSA/income/takeup 0.67 2012-04-06

UKMOD, for the same benefits, applies non-take-up probabilities sourced from DWP (2020) "Income-related benefits: estimates of take-up" and HMRC (2019) tax credits take-up, varying by claimant circumstance (UKMOD Country Report 2023–2030, CeMPA WP 8/26, Table 3.4):

benefit UKMOD assumption
Housing Benefit 0.85 pension-age · 0.57 working-age in work · 0.96 working-age not in work
Income Support 0.89 without children · 0.92 with children
CTC + WTC 0.95 lone parents (non-London) · 0.73 couples with children · 0.32 WTC no children
Pension Credit 0.69 guarantee · 0.37 savings-only

So on four legacy benefits PolicyEngine assumes everyone entitled claims, where the published DWP/HMRC evidence says between 32% and 96% do, depending on who they are.

Why it matters for the scorecard

Two distinct consequences, and I want to be careful not to overstate either:

  1. A caseload or expenditure comparison for HB/IS/CTC/WTC is not comparing the same quantity. PolicyEngine is producing an entitlement count; DWP's administrative figure is a receipt count. Any lane scoring those against each other is measuring non-take-up and attributing it to the engine. This is exactly the failure GAP_REASONS and the divergence registry (#59) exist to name — but nothing currently names it.

  2. Calibration may be silently absorbing it. PolicyEngine reweights to DWP/OBR caseload and expenditure targets. If entitlement is simulated at 100% take-up and then weighted to match a receipt-based total, the weights are compensating for a modelling assumption. The scorecard already distinguishes this — OBR rows carry calibration_relationship = consumed_as_target — but the take-up axis isn't represented at all.

Proposed lane

A parameter-level comparison rather than an output one, which the scorecard has none of today: record PolicyEngine's take-up parameters read live from the certified engine, alongside UKMOD's published assumptions, with the originating DWP/HMRC source cited for each.

It stages no value claims (the #86/#91 rule — the rates originate with DWP and HMRC, and UKMOD's table is a transcription of them plus mid-point choices of its own). What it records is: what each model assumes, where the assumption came from, and how stale it is.

Caveats worth stating up front

  • UKMOD applies take-up at household level per benefit with a randomisation rule; PolicyEngine's are national scalars. These are not the same mechanism, so the numbers are not strictly like-for-like — the comparable object is the assumed aggregate rate, not the algorithm.
  • PolicyEngine does have reported-recipient anchoring in policyengine-uk-data, so real behaviour may be closer than the bare parameter suggests. That should be checked before any conclusion is drawn about output error.
  • UKMOD flags its own rates as stale: DWP/HMRC have not updated working-age take-up statistics since 2021, and CeMPA says "we have, as yet, failed to identify a suitable remedy." PolicyEngine's are older still, and carry no such flag.

Blocked on nothing. Needs no compute (#128) and no new harvest.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the scorecard's GAP_REASONS, divergence registry, and existing lane or parameter-comparison entry points. Review how calibration_relationship and source citations are represented, then record the certified PolicyEngine take-up parameters beside UKMOD's published assumptions. Done means the comparable aggregate rates, originating DWP/HMRC sources, dates, and mechanism caveats are represented without making value claims.

Written by the indexing model from the issue text.

Assessment

Domain
analytics, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.