PolicyEngine / PolicyEngine/microcosm
Audit and gate UK formula-owned exports before publishing populace-uk
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Summary
The UK side needs its own tracker for the same export-class problem found in populace-us: non-structural PolicyEngine variables with formulas can be present in a saved H5 and then act as pinned inputs. The UK audit should be aggregate-first and assertion-based; do not silently drop columns.
Local audit context
A local populace-UK candidate showed formula-marked PE-UK columns present in the saved artifact. Broad removal is not safe without classification:
- Dropping
employment_income,capital_gains, oremployee_pension_contributionsmakes those aggregates zero, so those are currently acting as source/compatibility layers rather than harmless computed outputs. - Dropping
student_loan_repaymentsprevents the UK simulation from building during dataset uprating. - Several formula paths shift materially when recomputed, e.g.
rail_subsidy_spendingabout +11%,is_marriedabout +57%, enhanced disability flag about -88%, severe disability flag about -40% in the local one-column audit. - Some head flags appeared aggregate-equivalent, but they still should not be auto-dropped without a declared decision.
Fix direction
- Add a PE-UK counterpart to the formula-owned export review/gate.
- The gate should assert/report formula-owned exported columns; upstream build stages must make explicit keep/drop/input-surface decisions.
- Before changing any UK formula/input surface, compare stored aggregate vs formula-recomputed aggregate and record the result.
- Distinguish true computed outputs from PE-UK compatibility/source-input formulas that may need model-side or data-side handling.
- Add regression tests so a future UK export cannot silently mutate or drop formula-owned columns.
Related: #18, #24, #25.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the existing populace-us formula-owned export review/gate, then read related issues #18, #24, and #25 for the UK context. Define the PE-UK audit around aggregate-first comparisons and explicit keep/drop/input-surface decisions. Done means formula-owned exports are asserted or reported, stored and recomputed aggregates are compared, and regression coverage prevents silent column mutation or removal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100