PolicyEngine / PolicyEngine/microcosm

Persist structured target fields in the US target surface (geography/source/variable/breakdown)

Open
#57 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Problem

The US calibration target surface (target_surface_raw.npz) persists each target only as a flattened slash-joined string name, under three inconsistent conventions:

  • state/<source>/<variable>/<STATE_ABBR> — e.g. state/bea/wages_and_salaries/AL
  • nation/<source>/<variable>[/<dims>] — e.g. nation/irs/adjusted gross income/total/AGI in 10k-15k/taxable/All
  • US<fips>/<metric> — e.g. US01/snap-cost

Any consumer that wants to group, filter, or display targets by criteria (geography, source, variable, breakdown) has to re-parse these strings. The populace.dev /calibration page does exactly this today (scripts/build_dashboard_data.py: parse_target) — brittle and convention-dependent, and it loses information where the conventions disagree (e.g. US<fips> rows carry no source).

The target registry already knows the structured fields at construction time; the surface just drops them on export.

Ask

Have the exported target surface carry structured per-target fields alongside (or instead of) the flattened name — geography (+ level), source, variable, breakdown — mirroring the long-format schema the policyengine-us-data calibration targets already use:

DATA_SOURCE, GEO_ID, GEO_NAME, VARIABLE, VALUE, IS_COUNT, BREAKDOWN_VARIABLE, LOWER_BOUND, UPPER_BOUND

Acceptance criteria

  • The exported target surface includes structured per-target fields (geography + level, source, variable, breakdown) with a stable, documented schema.
  • A consistent geography key across the three conventions (national / state-abbr / state-FIPS).
  • Diagnostics consumers (populace.dev /calibration, the observatory) read the structured fields directly — no slash-string parsing in the consumer.

Related

  • #10 — publish calibration diagnostics (per-target rows, loss trajectory, skipped targets). The populace.dev /calibration page now realizes this, parsing the slash names as a stopgap until this issue lands.
  • #48 — US geography and local-area source support.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the target registry and the exporter that produces target_surface_raw.npz, then inspect scripts/build_dashboard_data.py and its parse_target stopgap. Define and document structured geography/level, source, variable, and breakdown fields with one geography key, and update the named diagnostics consumers to use them directly. Done means the exported schema is stable and slash-string parsing is gone.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.