PolicyEngine / PolicyEngine/microcosm
Drop raw-source and build-intermediate columns from the published populace-us artifact
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Summary
The published populace-us artifact should store PolicyEngine-recognized inputs plus explicit structural/geo columns, not raw source-survey fields or build scratch columns.
Drop (raw source + build intermediates)
- Raw CPS ASEC columns:
A_AGE,A_FTPT,WSAL_VAL,SEMP_VAL,WC_VAL,DIS_VAL1, etc. - Raw SCF columns such as
scf_net_worth,scf_bank_account_assets, and otherscf_*source fields once the PE inputs they feed are set. - Build scratch:
_half,_orig_household_id,*_is_puf_clone,new_tax_unit_id, and redundant intermediate weights if they duplicate calibrated weights.
Keep
- Geo identifiers used by the local-area pipeline:
block_geoid,tract_geoid,congressional_district_geoid,cbsa, etc. - Structural keys and calibrated weights.
Acceptance criteria
- The export surface has an explicit structural/geo allow-list.
- Raw source and scratch columns are removed upstream before final artifact writing.
- A gate fails when an artifact contains columns outside PE variables plus the allow-list.
- Tests cover a raw-source column, a scratch column, and a retained geo/structural column.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the populace-us export and the upstream final-artifact writing path, then trace where raw CPS/SCF fields and build scratch columns enter the pipeline. Add the explicit structural/geo allow-list and artifact gate, and run the relevant export tests to verify raw-source and scratch columns are rejected while a retained geo or structural column remains.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100