PolicyEngine / PolicyEngine/microcosm
Carry manifest rewrites into the shared Stage/StageRecord producer graph
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Found by the Codex adversarial review of #717 (medium; dispositioned as a deferral in the PR's review-response comment).
Mechanics
country_stage_plan compiles only stage.outputs into Stage.produces (packages/microcosm-build/src/microcosm/build/country_spec.py, plan-assembly block); the parsed rewrites key (source_manifest.py:202) never reaches StagePlan or StageRecord. A stage that rewrites already-produced columns — E3's frs_education_grant_split (housing_benefit_reported), and now E7's hmrc_spi_income_spine (38 rewrites: the six income components, the 29 stage-2 columns, the three retained leaves redrawn on SPI rows) — can materially change final column values while:
StageRecord.producedand per-stagenonzero_sharestill attribute the column to its original producer;- the driver's
--emit-nonzero-shares"final" list is built fromrecord.produced(plus a hardcodedFRS_EDUCATION_GRANT_REWRITEScarve-out intools/build_uk_frs_spine.py— evidence the gap has already forced one ad-hoc patch); - coverage/provenance evidence points at the pre-rewrite producer.
The declarations themselves are reviewable and lockstep-tested (E7's test_e7_outputs_and_rewrites_are_backed_by_runtime_constants pins them against runtime written-column constants), so this is a provenance/reviewability gap, not a correctness hole — but it weakens the one-canonical-producer story exactly where stages intentionally overwrite.
Proposed shape
- Carry
rewritesas first-class metadata onStageandStageRecord. - Validate at plan assembly that every rewrite has an earlier canonical producer in the selected plan (fail closed on rewrites of never-produced columns).
- Record rewrite-stage provenance (and per-stage nonzero shares for rewritten columns) separately from newly produced columns, and derive the drivers' "final columns" lists from produces ∪ rewrites instead of hardcoded carve-outs.
Shared-layer change (US/BE unaffected behaviorally today — no US stage declares rewrites — but the validation applies country-agnostically).
Refs #717 (adversarial review), #683/#145 (E7/WS-E), #679 (E3's grant-split precedent).
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the plan-assembly block in packages/microcosm-build/src/microcosm/build/country_spec.py and the parsed rewrites at source_manifest.py:202. Trace how Stage and StageRecord expose produced columns, then inspect tools/build_uk_frs_spine.py and the existing E3/E7 tests. Done means rewrites are carried through the shared metadata, validated against earlier producers, represented in provenance and nonzero-share reporting, and no hardcoded carve-outs remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, build-system, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100