PolicyEngine / PolicyEngine/microcosm

Carry manifest rewrites into the shared Stage/StageRecord producer graph

Open
#721 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Found by the Codex adversarial review of #717 (medium; dispositioned as a deferral in the PR's review-response comment).

Mechanics

country_stage_plan compiles only stage.outputs into Stage.produces (packages/microcosm-build/src/microcosm/build/country_spec.py, plan-assembly block); the parsed rewrites key (source_manifest.py:202) never reaches StagePlan or StageRecord. A stage that rewrites already-produced columns — E3's frs_education_grant_split (housing_benefit_reported), and now E7's hmrc_spi_income_spine (38 rewrites: the six income components, the 29 stage-2 columns, the three retained leaves redrawn on SPI rows) — can materially change final column values while:

  • StageRecord.produced and per-stage nonzero_share still attribute the column to its original producer;
  • the driver's --emit-nonzero-shares "final" list is built from record.produced (plus a hardcoded FRS_EDUCATION_GRANT_REWRITES carve-out in tools/build_uk_frs_spine.py — evidence the gap has already forced one ad-hoc patch);
  • coverage/provenance evidence points at the pre-rewrite producer.

The declarations themselves are reviewable and lockstep-tested (E7's test_e7_outputs_and_rewrites_are_backed_by_runtime_constants pins them against runtime written-column constants), so this is a provenance/reviewability gap, not a correctness hole — but it weakens the one-canonical-producer story exactly where stages intentionally overwrite.

Proposed shape

  1. Carry rewrites as first-class metadata on Stage and StageRecord.
  2. Validate at plan assembly that every rewrite has an earlier canonical producer in the selected plan (fail closed on rewrites of never-produced columns).
  3. Record rewrite-stage provenance (and per-stage nonzero shares for rewritten columns) separately from newly produced columns, and derive the drivers' "final columns" lists from produces ∪ rewrites instead of hardcoded carve-outs.

Shared-layer change (US/BE unaffected behaviorally today — no US stage declares rewrites — but the validation applies country-agnostically).

Refs #717 (adversarial review), #683/#145 (E7/WS-E), #679 (E3's grant-split precedent).

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the plan-assembly block in packages/microcosm-build/src/microcosm/build/country_spec.py and the parsed rewrites at source_manifest.py:202. Trace how Stage and StageRecord expose produced columns, then inspect tools/build_uk_frs_spine.py and the existing E3/E7 tests. Done means rewrites are carried through the shared metadata, validated against earlier producers, represented in provenance and nonzero-share reporting, and no hardcoded carve-outs remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, build-system, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.