PolicyEngine / PolicyEngine/microcosm

Rebuild the US ACS local-area artifact on the certified Build O lineage (buildl 36de5d9a is stale)

Open
#512 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Why

The only local-area product in circulation is populace-us-2024-buildl-acs-local-36de5d9a-20260712T104640Z (built 2026-07-12 from branch buildl-acs-spine @ 32d446b, donor = the June dense base). It predates every target-surface repair of the Build M/N/O campaign:

  • capital-gain-distributions defect era (the +635% signature; pre-memo-leg, pre-truthful CG/interest chains),
  • pre-#469/#488 truthful-surface work and the #491 shared critical-target register,
  • old SSI machinery (pre one-shot Bernoulli seeding),
  • no adult-care / SE-health inputs (#451 items 1–2, landed in #500),
  • a donor that cannot satisfy the 163-column input-coverage surface base-O now proves live.

policyengine.py's dataset_overlays.us.populace_us_2024_acs_local still pins this stale artifact (sha 36de5d9a…). The QRF transfer replicates the donor distribution onto the ACS spine, so every donor-side repair since July 12 is absent from the only local-area surface consumers can name.

Plan

Rebuild the ACS local artifact on the certified Build O lineage — base-O (base_populace_us_2024_puf_support.h5, sha 356cf501e1ca…, 23 stages, block/tract/CD geography assigned) — the same lineage as the certified default populace-us-2024-buildo-sparse-rmloss100-22bd902-20260722T232627Z.

  1. Port the Build L tooling into tools/ as first-class code (it lives on the unmerged buildl-acs-spine branch + _buildl-runtime scripts today): acs_pums / acs_sources / acs_inputs / acs_transfer / acs_multispine / base_pool modules with their test suites, the multispine staging driver, and the materialize→calibrate→finalize→package chain — so the next refresh is one command.
  2. Donor semantics updated for the 23-stage base: the QRF plan's PUF-detail families auto-extend from the current constants; explicit families extend to the new donor-observed inputs (adult-care, SE-health); buildl-era exclusions (partnership_self_employment_net_earnings, s_corp_income) re-verified against base-O.
  3. Geography: donor rows keep base-O's certified block-ladder assignment (block/tract/CD/county/place/SLD/CBSA) instead of buildl's state-conditional PUMA re-draw; donor puma derives deterministically from tract_geoid via the PUMA ladder's tract-overlap table (2020 tracts nest in exactly one 2020 PUMA). ACS rows keep buildl semantics: observed PUMA → seeded CD/county assignment.
  4. Calibration surface unchanged (no new knobs, #492 embargo): state+CD population marginals + state SNAP benefits/caseloads + state Medicaid enrollment (RI substitution) + state SOI, compiled from the current feed vintage (v9.3, eab45a2b…); 800 epochs / 400-epoch warm-start batches / ratio 5.0 / lambda 0 / seed 0 / mass conserve — the Build L runbook decisions verbatim.
  5. Publish per the local product's contract: implement #398's dataset-role classes (non_default_local_area gets its own validation: gates object + pass, diagnostics shape, source coverage keyed to the families actually calibrated, reviewed-limitations register; pointer ops forbidden), then publish through the tracked path — immutable tag on policyengine/populace-us, latest.json untouched.
  6. policyengine.py overlay repin PR: new revision + sha256 in dataset_overlays.us.populace_us_2024_acs_local; the overlay-preserved / default-resolution-unchanged semantics tests stay intact.

Known boundaries the release will declare

  • #507 SSI aged-band collapse, inherited by construction — base-O carries the 65+ take-up banding defect (aged baseline 0.94M vs SSA 2.42M). This rebuild does not wait for the fix: the committed one-command chain makes the O-2-based refresh mechanical once #508's fix certifies.
  • #403's buildl QA findings re-measured, not assumed fixed: per-spine SSI incidence, weight-composition skew, and ESS get measured on the new artifact and recorded in its diagnostics.
  • #393 miscellaneous_income loss-side donor defect (open): restated as inherited via transfer if still measured on this lineage.
  • CD population marginal vintage 2020 (honest v1); ACS-spine GQ-housing structural nulls; mixed-coverage sub-PUMA columns (donor rows carry block/tract, ACS rows do not); ESS concentration under the hard 5× cap with lambda 0.

Receipts

Build receipts land here and on #449 (the Build O battery thread). Relates #48, #398, #403, #449, #492, #507.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unmerged buildl-acs-spine branch, its _buildl-runtime scripts, and the tools/ modules named in the plan. Trace the multispine staging driver and materialize→calibrate→finalize→package chain, then inspect policyengine.py's dataset_overlays.us.populace_us_2024_acs_local. Done means a validated Build O artifact, published tracked metadata, and an overlay repin with the stated semantics tests intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, data-engineering, release, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.