PolicyEngine / PolicyEngine/policyengine-uk-data

ETB services imputation divides household totals by household size and stores the per-head figure in household-entity columns

Open
#469 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
2
Avg merge
1d 14h
Merged PRs (30d)
4

Description

Title: ETB services imputation divides household totals by household size and stores the per-head figure in household-entity columns

Summary

imputations/services/etb.py derives household-level service values from the ETB donor and then divides each household's total by household size before storing the result — but the columns it writes are household-entity columns. Every downstream consumer that reads them as household totals (weighted aggregation, calibration, distributional analysis) therefore under-counts by a factor of mean household size on the affected surfaces.

This is the second defect of the family documented in #467 (the Scottish water fill-after-addition NaN-zeroing found during the microcosm spine-parity work): a stage-internal unit convention leaking into the stored entity grain.

How it was found

During the microcosm#686 whole-spine parity comparison (2026-08-24/25), the ETB services surfaces diverged from the spine's donor-faithful port. Measuring the donor directly (the stage's own cleaning function over the pinned tab, survey-weighted — the measurement doctrine from that workstream) showed the incumbent's stored per-household levels sitting far below the donor's household totals, consistent with a per-head division; inspection of imputations/services/etb.py confirmed the division site.

What it does and does not explain

  • It does explain part of the systematic level shortfall on the ETB service columns relative to the donor.
  • It does not reconcile the levels on its own: rail subsidy, for example, remains ~3.07× the donor mean even after accounting for the division. So this is recorded as a defect observation, not a complete explanation of the ETB level divergences — there is at least one further mechanism in the chain.

Suggested fix

Either store genuinely per-head values in person-entity columns, or keep household totals in the household columns — pick one convention and make the variable names say which. If the per-head figure is intentional for some consumer, that consumer should divide at read time rather than the store carrying a mixed-grain surface.

Context on urgency

The microcosm migration (microcosm#665) replaces this pipeline; the spine's ETB port is donor-faithful and does not reproduce the defect, and the spine-vs-incumbent difference is signed in microcosm's parity register. Filing so the incumbent's remaining release window doesn't ship the defect unrecorded, and so the #467 family has both members documented in one place.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in imputations/services/etb.py at the household-size division and trace the columns it writes through downstream consumers. Compare the stored values with the ETB donor household totals, then make the entity grain consistent and verify that weighted aggregation, calibration, and distributional analysis no longer under-count affected surfaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.