PolicyEngine / PolicyEngine/microcosm

UK output-area-anchored geography ladder (US block-ladder pattern)

Open
#349 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

UK output-area-anchored geography ladder (the US block-ladder pattern for the UK)

Goal

Give the UK artifact the same local-geography spine the US artifact has (us_runtime/geography_ladder.py): anchor every household at the finest published census geography, derive every other layer deterministically from that anchor, and keep one national dataset, filterable at any grain (the #275 standing rule — no per-area files).

Design (mirroring the US block ladder)

Anchor = census output area (OA). The UK's block equivalent: 2021 Census OAs for England & Wales, Scotland's Census 2022 OAs, NI small areas. Assignment ladder:

  1. Region is already assigned (FRS carries it; it is a calibrated dimension of the national build).
  2. Assign a Westminster parliamentary constituency (2024 boundaries) within region, sampled proportional to constituency household counts — the politically salient rung, and the rung where per-area calibrated weights (#146 long weights) can later bind, exactly as the US anchors its ladder inside the assigned congressional district.
  3. Assign one OA within the constituency, sampled proportional to OA population (ONS OA→PCON24 best-fit lookup defines membership).
  4. Every other layer derives from the OA — no independent randomness:
    • LSOA and MSOA: structural nesting (OA → LSOA → MSOA).
    • Local authority district: ONS OA→LAD lookup.
    • Ward: OA→ward lookup.
    • ITL levels / combined authorities: LAD-level lookups.

UK geographies famously do not nest cleanly (constituencies cross LAD boundaries); the ONS Open Geography Portal best-fit lookups are the sanctioned resolution and should be carried in the ladder artifact itself, sha-pinned, like the US ladder artifact carries its block crosswalks.

Discipline ported from the US implementation

  • Vintage policy = error: the ladder artifact records one vintage per derived layer; the loader refuses an artifact missing any; assignment refuses a ladder whose constituency vintage differs from the vintage the constituencies were assigned under (the #205 lesson — no silent partial joins on mismatched vintages).
  • Engine inputs only: exported columns are policyengine-uk household inputs (or plain data columns where no engine input exists); never persisted formula outputs (the #34 lesson — derived quantities recompute from the rung).
  • Seeded draws: constituency and OA sampling under the build's seed discipline, reproducible from the release manifest.

Relationship to the existing UK local machinery

Complementary, not competing, with #146/#147:

  • The ladder gives the spine: every household locatable at any grain, local analysis = filter (or aggregate) the national file.
  • The long weights (#146, local_solver/local_runner, already ported) give calibrated per-area estimate quality on top of the spine where area targets exist.
  • The credibility gates (#147) score whether a given area's filtered/weighted estimate is publishable — random-within-region assignment is only as good as region-level joint structure, and the gates are what say so per area.

Acceptance

  • Ladder artifact (OA anchor + lookups + vintages) built from ONS sources, sha-pinned, with provenance in the ledger.
  • Assignment wired into the UK build behind the country-spec geography-spine schema; gate output mirrors us_geography_ladder_assignment_summary.
  • National artifact round-trips: filter by constituency/LAD/region reproduces the calibrated region totals; assignment summary reports per-rung coverage.
  • Cross-checks: constituency household counts vs ONS, LAD population shares vs mid-year estimates (report-only at first; gates per #147).

First milestone: England & Wales ladder end-to-end on the current populace-uk 2023 artifact; Scotland/NI rungs follow once their lookup vintages are pinned.

Refs: #146 (local variant publication), #147 (credibility gates), #145 (source refresh), #275 (one-national-dataset rule), populace#8 / policyengine.py#427 (the artifact this extends).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/populace-build/src/populace/build/us_runtime/geography_ladder.py, then trace the UK build's country-spec geography-spine schema and the existing local_solver/local_runner machinery. The first milestone is an England and Wales ladder for the current populace-uk 2023 artifact; done means the sha-pinned artifact, assignment summary, national round-trip, and stated cross-checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.