PolicyEngine / PolicyEngine/microcosm

County-calibrated County Files (3,144 counties, 2026 + 2027): build epic

Open
#922 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

County-calibrated US files for all 3,144 counties, years 2026 and 2027, delivered as one HDF5 per year with the six entity tables. Scope committed externally; MVP target is 6–10 weeks from the agreement's effective date (mid-September 2026). This epic collects what the repo has and what it lacks, verified on 2026-09-03 against origin/main and the installed engine.

What exists

  • Income-blind county assignment on both geography lanes: us_runtime/puma_ladder.py (county sampled within PUMA by block-population overlap) and us_runtime/geography_ladder.py (block draw within CD; county_fips = block_geoid[:5]). Both write county_fips, sldu, sldl, cbsa_code, etc. onto the household table.
  • Occupation already on every person record (detailed_occupation_recode, org_wages stage).
  • Engine inputs for industry recodes and worked-last-year merged in policyengine-us #9311; the pin bump landed in #756.
  • Writers are period-parametric (write_dataset(..., period=), write_nullable_us_h5(..., period=)).

What is missing

  1. County target surface and loss rung. packages/microcosm-calibrate/ has no county target family, no county fact ingestion, no county rung; the ACS-local tool calibrates state + CD only (tools/build_us_acs_local_release.py column_map). Needed: ACS 5-year county income tables (B19001 / B19013 / S2001), IRS SOI county, Census county population; Ledger ingestion; multi-level consistency national → state → county.
  2. Support. Pooled base ≈ 338k households over three ASEC years, ~107 per county on average before L0 pruning. Decide pool scale and pruning policy for county grain.
  3. 2023 county geography. The block ladder is built on 2020 tabulation blocks (block_vintage: 2020_tabulation_blocks), so Connecticut resolves to the eight legacy counties (3,143 areas). The commitment is 2023 geography with the nine planning regions (3,144). Needs a 2020-block → 2023-county crosswalk artifact and a re-cut ladder.
  4. Non-2024 periods. PERIOD = 2024 is hard-coded in tools/build_us_fiscal_refresh_release.py and tools/build_us_acs_local_release.py; no US release has been produced at another period. 2026 needs aged targets (us_runtime/target_aging.py exists; confirm production wiring) and 2027 is that again on projected targets.
  5. Industry and worked-last-year on the records. #745 (work_experience_inputs stage; parent #719) is still a draft with no CI; it needs a rebase onto post-#756 main, bundle/ABI regeneration, and a regenerated ASEC pooled base since the frozen census_cps inputs lack WEIND/WEMIND.
  6. Household measures as materialized values. household_net_income and household_market_income are formula-owned (adds/subtracts), and the export gate refuses formula-owned columns in entity tables (adapters/policyengine_us.py, "Export contract violated"). The delivery commitment is earnings, gross income, and net income as materialized values keyed by household_id, readable without running the engine: ship a derived-measures table alongside the entity tables (release-directory sidecar or a non-input HDF5 key) computed at the delivery period.
  7. weeks_worked at 2026/2027. The engine defines formula_2025, so the export guard drops it from the person table at 2026+. Documented to the partner already; make sure schema docs list it.
  8. Schema documentation emitter. Release contract files today (build_manifest, calibration_diagnostics, us_source_coverage, gate_summary, held_back_columns, ...) include no column-level schema doc. Each delivery needs a table/column/dtype manifest plus a diff (additions, renames, removals) against the prior release; first known entry is would_claim_wictakes_up_wic_if_eligible.

Sequencing

  1. Rebase and land #745; regenerate the pooled base and multispine pool with the new columns.
  2. Build the county target surface and the 2023 geography crosswalk in parallel.
  3. Calibrate at county grain with per-county support/ESS diagnostics and multi-level gates; publish a validation report per county (record count, calibrated aggregates vs targets, deviation).
  4. Produce the 2026 file, then 2027 on projected targets.
  5. Derived-measures sidecar + schema manifest emitter; documentation of file layout.

Related: #719, #745, #696, #292, #367.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sequencing section, then inspect packages/microcosm-calibrate/, us_runtime/puma_ladder.py, us_runtime/geography_ladder.py, and the two release tools named in the issue. Review #745 and its related issues before choosing a narrowly scoped component. Done requires the county targets, geography, period handling, materialized measures, schema manifest, calibration outputs, and 2026/2027 deliveries to satisfy the stated contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.