PolicyEngine / PolicyEngine/microcosm
Tighten the UK input-mass tolerance by receipting the explained top drifts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Raised in review of #706 (vahid-ahmadi): the UK input-mass parity gate's shared relative_tolerance is minted honestly as the worst surviving measured drift, but that edge is currently set by a column whose deviation is already explained — so the fence grants every other column a much larger unexplained-regression budget than the compared population actually needs.
The measurement (full surface, recorded in experiments/630-uk-gate-adjudication-receipts.md)
128 columns compared against the efrs-post-calibration reference at floor 0.0. The compared population sits at median 0.24, p75 0.42, p90 0.63, p95 1.15 |relative drift|, while the committed tolerance is 4.5218. Only five columns exceed 1.5. So the gate today is a gross mass-loss/explosion fence, not a regression detector: a 2× mass move in a mid-band column passes silently.
The decay above the bulk is smooth, with no natural cut point — ranks 2–8: is_enhanced_disabled_for_benefits 4.5218, sda_reported 3.0587, jsa_income_reported 2.1690, lump_sum_income 1.5237, education_grants 1.1557, bsp_reported 1.1509, non_residential_property_value 0.9026.
| columns receipted into the register | re-minted tolerance | tightening |
|---|---|---|
1 (today: charitable_investment_gifts only) |
4.5218 | — |
| 2 | 3.0587 | 1.48× |
| 4 | 1.5237 | 2.97× |
| 7 | 0.9026 | 5.01× |
| 20 | 0.5167 | 8.75× |
| 30 | 0.4401 | 10.27× |
Why this was deferred out of #706
The instrument already exists there — per-reference reviewed exclusions with receipts, expiry, and stale-fail — and re-minting the tolerance is a parameter edit under the same reviewed mechanism. What is missing is evidence per column, and the register deliberately refuses unevidenced reasons (reason, approved_by, adjudication, approved_on, expires_on).
Today exactly one of the top drifts carries a documented explanation: #703's head-to-head receipt verified the disability categories and flags agree row-for-row at 100.0000% with the incumbent's own create_frs at the pinned revision, attributing the residual delta against the frozen reference to the artifact's vintage (rev 655dd07e predates the disability-logic fixes) plus the SPI-synthetic composition (~37% of persons), with re-measurement scheduled at matching composition at E10. Excluding only that column re-mints at 3.0587 — a 1.48× gain that does not change the gate's practical sensitivity.
sda_reported and jsa_income_reported are plausibly the same compositional story (FRS-reported legacy-benefit columns under a shifted benefit population), but carry no verification of their own.
The work
- For each of the top ~7 drifts, establish whether the deviation traces to the known comparability causes (reference vintage; SPI-synthetic composition) or is a genuine candidate-side deviation — the #703 head-to-head is the methodological template.
- Receipt the explained ones into
input_mass_reviewed_exclusions.jsonunderefrs-post-calibration, expiring at the E10 recomposition so the register is forced back open exactly when the composition story changes. - Re-mint
relative_toleranceat the worst unexplained survivor, with the receipt naming the surviving edge column. - Consider whether an explained-drift entry should be a distinct register class from a suppression entry — an exclusion removes all regression detection for that column, which is a real cost the current single-class register does not express.
Note the asymmetry in (4): every column moved into the register trades "no detection at all on that column" for "better detection on the remaining 127". That tradeoff, not just the tolerance number, is what the triage should decide.
An alternative worth weighing at the same time: per-column tolerance bands (most precise, ~128 parameters, high re-mint churn) versus the shared tolerance plus register (coarse, cheap, what exists today).
Refs #706 (review thread and the receipts doc), #703 (the receipt precedent), #609/#610 (the measurement doctrine), #665 (E10 recomposition).
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with experiments/630-uk-gate-adjudication-receipts.md and #703's head-to-head receipt, then inspect input_mass_reviewed_exclusions.json and the reviewed-exclusion mechanism from #706. For each top drift, establish whether the documented comparability causes explain it and record supported exclusions with E10 expiry. Done means the tolerance is re-minted at the worst unexplained survivor and the register-class or per-column-band tradeoff is decided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100