PolicyEngine / PolicyEngine/microcosm
Persist structured target fields in the US target surface (geography/source/variable/breakdown)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Problem
The US calibration target surface (target_surface_raw.npz) persists each target only as a flattened slash-joined string name, under three inconsistent conventions:
state/<source>/<variable>/<STATE_ABBR>— e.g.state/bea/wages_and_salaries/ALnation/<source>/<variable>[/<dims>]— e.g.nation/irs/adjusted gross income/total/AGI in 10k-15k/taxable/AllUS<fips>/<metric>— e.g.US01/snap-cost
Any consumer that wants to group, filter, or display targets by criteria (geography, source, variable, breakdown) has to re-parse these strings. The populace.dev /calibration page does exactly this today (scripts/build_dashboard_data.py: parse_target) — brittle and convention-dependent, and it loses information where the conventions disagree (e.g. US<fips> rows carry no source).
The target registry already knows the structured fields at construction time; the surface just drops them on export.
Ask
Have the exported target surface carry structured per-target fields alongside (or instead of) the flattened name — geography (+ level), source, variable, breakdown — mirroring the long-format schema the policyengine-us-data calibration targets already use:
DATA_SOURCE, GEO_ID, GEO_NAME, VARIABLE, VALUE, IS_COUNT, BREAKDOWN_VARIABLE, LOWER_BOUND, UPPER_BOUND
Acceptance criteria
- The exported target surface includes structured per-target fields (geography + level, source, variable, breakdown) with a stable, documented schema.
- A consistent geography key across the three conventions (national / state-abbr / state-FIPS).
- Diagnostics consumers (populace.dev
/calibration, the observatory) read the structured fields directly — no slash-string parsing in the consumer.
Related
- #10 — publish calibration diagnostics (per-target rows, loss trajectory, skipped targets). The populace.dev
/calibrationpage now realizes this, parsing the slash names as a stopgap until this issue lands. - #48 — US geography and local-area source support.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the target registry and the exporter that produces target_surface_raw.npz, then inspect scripts/build_dashboard_data.py and its parse_target stopgap. Define and document structured geography/level, source, variable, and breakdown fields with one geography key, and update the named diagnostics consumers to use them directly. Done means the exported schema is stable and slash-string parsing is gone.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100