PolicyEngine / PolicyEngine/microcosm

UK graph: move geography assignment ahead of enrichment (geography-first reordering)

Open
#931 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Why. The release path (docs/us-uk-release-path.md) fixes one construction order for both countries: complete multispine including its initial clones → assign one atomic area per household → derive larger geographies from versioned mappings → enrichment that retains the assigned location → calibration → full and compact exports. The UK full-build graph registered by #901 keeps the older UK order: every spine stage runs on the source FRS before cloning, the K geographic clones are created after the spine, and the legacy ladder draw assigns geography just before target compilation. That draw is not keyed to household identity (row order or K changes every draw), the ladder carries no machine-readable exact / best-fit relation per layer, and its publisher provenance is only partly committed. This issue tracks the PRs that move the UK to the shared order and the shared operators, each with its own before/after evidence.

Fixed decisions (2026-09-14/15). Atomic level stays the lowest published census unit per nation: E&W 2021 Output Areas (188,880), Scotland 2022 Output Areas (46,363), NI 2021 Data Zones (3,780). Sampling law becomes a single-stage draw of the atomic area by census household count within the household's FRS region. Vintages for FY2024-25: LAD April 2023 code set (the 1 April 2025 Barnsley/Sheffield recode is item (e) below), ITL 2025 for all nations, Ward 2024, Westminster constituencies July 2024. K, seed and solve doctrine are unchanged by any PR in this list. The UK geography.yaml stays phase: legacy until item (f).

Tracked PRs (checklist).

  • (a) Assignment swap (step 2; stacked on #901; draft PR #932): replace uk.full.locations / uk.full.geography_mapping with the shared atomic_geography_nodes (support import, assign, derive, gate) keyed by geography_household_key from the existing household_draw_key lineage; single-stage law in the UK adapter; three support artifacts rebuilt from published lookups with a committed provenance JSON and sources.yaml entries; --geography-assignment legacy kept for measurement builds only (--release-candidate refuses it); assignment-level before/after on {legacy two-stage, keyed single-stage} × {K=1, K=15} with per-area evidence (887 format). No calibration A/B.
  • (b) Enrichment reorder, one area-dependent stage at a time (frs_brma, was_wealth, lcfs_consumption, regional_uprating, council tax), each with a distribution receipt against the pre-move build; decide per stage whether it conditions on assigned local authority, constituency or region. Weight-changing stages (SPI support channel, CGT clones) must stay before the origin-budget freeze point.
  • (c) Origin budget and caps: decide the budget's reference weight once (b) settles how weights flow into calibration; decide whether the UK calibrated node adopts the US two-cap declaration (solver bound against incoming importance weights plus the executor's design cap). No solve-doctrine change lands with (a) or (b).
  • (d) Rosters and area support from the supports: derive the 650/361 roster and the terminal area-support summary from the three support artifacts; retire the ladder NPZ and local_area_crosswalk.json's ladder pins.
  • (e) LAD 2025 recode: vintage translation for Barnsley E08000016→E08000038 and Sheffield E08000019→E08000039 in the #205 pattern, when 2025-coded facts arrive in Chronicle.
  • (f) F3 geography.yaml bundle diff: move the UK spec from phase: legacy to phase: f3 declaring the OA anchor, single-stage draw, shared kernels, per-layer vintages, typed source refs for the three supports and a keyed seed stream; needs two shared spec-engine extensions (a support list field in geography.schema.json, a keyed seed site in seeds.py), reviewed with Max.
  • Chronicle: issue PolicyEngine/chronicle#269 and its raw-only registration PR for the sixteen publisher lookup files (separate lane).

Out of scope for this issue. Calibration doctrine, K, the UC element and Chronicle schema work (#882, chronicle#259/#261/#262), and the US pipeline.

Evidence. Each PR records its receipts under experiments/931-… and per-cell evidence under docs/evidence/uk-931/, with sha256s in the PR body. Nothing here certifies a release; the release battery and holdout stay separate gates.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with draft PR #932 and the existing UK full-build graph from #901, then inspect the shared atomic_geography_nodes operators and the geography_household_key lineage. Read docs/us-uk-release-path.md, geography.yaml, geography.schema.json, and seeds.py, and review the experiments/931-* and docs/evidence/uk-931/ paths. Done requires the tracked stages, support artifacts, provenance, evidence receipts, and F3 specification changes to be completed without changing the stated solve doctrine.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.