Deliver a dense SPM-compatible US successor with verified hours and incumbent comparisons

Outcome

Produce a dense US data successor that works with the canonical spm-calculator integration, restores genuine usual-hours inputs for SNAP analysis, and demonstrates no unexplained material deterioration against the incumbent in national, state and congressional-district analysis.

The shortest candidate path repairs the existing dense population. Common-graph construction continues in #893; this scoped successor does not claim that the full new graph has completed or passed release gates.

Exact incumbent

• Build: populace-us-2024-buildp-acs-local-592ae5d6-20260819T020303Z in policyengine/populace-us.
• File: populace_us_2024_acs_local.h5, SHA256 ae1a46854de735614eff8e7708d667319a621d346c92dc3737969b91a1945bb7.
• 1,588,854 households and 3,589,209 people; full file identity and numeric counts checked.
• The consumer baseline includes country 1.825.2. Freeze the complete old/new runtime and analysis-year matrix before comparison; older historical audits are context, not fresh baseline results.
• Preserve household/person support, current identities, unaffected inputs and assigned geography. Any SPM membership changes require an explicit crosswalk and renewed outcome checks. Inherited calibration ancestry is not a new calibration verdict.

Current evidence and implementation

Component	Evidence/status
Native ACS hours mapping and source gate	#923; source-boundary classification and temporary-work follow-up fixed at f55b5da106a476134094ef878ececc61770b0377; 561 relevant tests pass; new CI running
Exact-parent ACS source recovery	#924; all 23 hosted checks pass at 3ae3dff0ef101774ce2a6d2d9fcc177f910c8c67; 27 synthetic cases, 64-household and 4,096-household pilots, then all 1,531,614 ACS households / 3,422,888 people
Corrected canonical unit assembly	PolicyEngine/spm-calculator#45; parent-link ordering and foster-age boundary
Country measurement universe and nullable outcomes	PolicyEngine/policyengine-us#9462; source/regression qualification, consumer integration still pending
Wrapper summaries and nullable JSON	PolicyEngine/policyengine.py#519; full hosted suites pass on Python 3.13 and 3.14 (968 passed, 4 skipped each); managed-bundle binding still pending
ACS partition adapter	47 synthetic fixtures plus 64-household and 512-household actual numeric grouping pilots pass against corrected canonical source; 512-household monetary regrouping also passes; country outcome checks pending
Age-15 hours donor preparation	#926; 26 synthetic tests and actual numeric preparation pass: 1,216 pooled donors, 39,800 recipients; output cells independently checked; no imputation yet

Recovery found a stored default of 40 hours on all 3,422,888 ACS people in the incumbent. Recovered source fields provide 1,767,132 observed usual-hours values and 1,105,858 source-confirmed past-year nonworkers; no hours cells for people aged 16 or older remain unresolved. The remaining source-universe exclusions are 39,800 people aged 15 and 510,098 younger children. The recovered sidecar is not a repaired or qualified dataset.

The full recovery checks two source-key joins plus every accessible retained numeric anchor under a documented sort/rank reconstruction. It does not invent an original staging revision, verify inaccessible PUMA values, or claim a new raw-ASEC join for the retained donor.

The first actual grouping pilot contains complete source households selected to exercise minor references/spouses/partners, unrelated children/adults, foster children and group quarters. It produces 72 proposed resource units from 64 incumbent units under the explicit development preset: eight residual separations and four child-pooling assumptions, with 48 GQ records retained. Five households retain secondary-relationship uncertainty in provenance; membership and role inputs are complete under the selected assumptions. Both minor-partner settings pass and preserve membership. This is a structural pilot, not a representative estimate, monetary regrouping or poverty calculation.

A broader case-enriched 512-household/1,183-person grouping pilot also passes both settings: 542 proposed units, 30 residual separations, seven child-pooling assumptions and 62 GQ people retained, with input/tax membership unchanged. Eight households retain source relationship uncertainty. The monetary-regrouping helper now passes 37 invented tests and independent source review, including a corrected canonical-label uniqueness check. It conserves positive childcare when exactly one plausible successor is the reference unit and retains ambiguous amounts in an exception ledger. Actual monetary application now passes: 21 old units split, all $4,800 of their childcare costs allocated exactly once, no unresolved amounts, and unchanged inputs. The alternate rent-free tenure classification changes ten successor units only. Original source/support/tenure strings are authenticated from the exact parent; all four inherited defaults match their original model source. This is still not a country outcome calculation.

A full independent scan of the pinned Census ACS archive confirms literal zero wages and self-employment income for all 39,800 people aged 15, despite blank usual-hours fields. That source pattern is not a local fill, but its cause remains unverified. The initial age-15 hours specification therefore excludes earnings. All 399 selected donors now match raw source keys, retained anchors, supplement weights and geography. Their source year means income year 2024 / March 2025 survey. In actual household-held-out validation, sex/region QRF performs worse than the unconditional survey-weighted benchmark in every fold on CRPS and Brier score; it is not adopted as superior.

The complete public ASEC age-15 cohort has also been recovered: 2,174 people in 2,083 households, 243 with positive hours, no zero weights. Its source-weighted mean is 2.059 hours, compared with 3.071 in the selected 399 donors. That difference supports using the full public donor cohort instead of assuming the retained support has the population distribution. Full-cohort household-held-out validation is complete: weighted empirical CRPS 1.97930 versus QRF 2.03207, and Brier 0.091718 versus 0.093151. QRF wins one fold and loses two; the simpler full-source empirical distribution is selected for development completion without claiming statistical significance or observed ACS outcomes. No recipient imputation or repaired H5 is claimed.

An independently reviewed synthetic development consumer passes real canonical assembly, country calculation and wrapper nullable output classes. A narrow compatibility PR, PolicyEngine/policyengine-us#9466 (9f9510b735b3b9b2e375854d66fd315202e31d35), permits the corrected calculator patch version while preserving the default lock. The combined development consumer resolves normally with no overrides and passes package consistency checks for all 85 packages. This closes that dependency conflict, not managed-bundle binding or native data acceptance. Country #9462 still has genuine default-data tests failing because older datasets lack the newly required universe declarations; those failures remain visible and the declaration requirement is not relaxed.

Country testing exposed and resolved a concrete dependency issue: the existing Microdf 1.2.1 lock counts excluded NaN observations in weighted denominators. The minimal explicit Microdf 1.3.0 update on country PR #9462, commit 468d5752d6ff470ab6da924c3e8ef417bf23d67c, keeps Core 3.30.2 fixed and passes all 112 selected cases: 54 focused and 58 existing SPM regressions. All 147 unrelated lock records are unchanged. This qualifies those tests on that exact dependency set; hosted CI and native consumer qualification remain separate. Preserve the earlier failure evidence rather than transferring acceptance between different dependency sets.

The exact incumbent calibration diagnostic artifact is now retained: 4,459 target rows, SHA256 b6f05b652049f88c044f1907eeed13c378aea47aec9c6dd6518faa90d1a0dcd0. These comprise 3,819 IRS targets, 153 SNAP/Medicaid targets, 51 state-population targets and 436 district-population targets. This is a historical comparison register, not a fresh run or a transferred pass. The old release already has material limitations, including low effective sample size and 56 IRS rows above 10% relative error; they remain visible.

For initial SPM diagnostics, an explicitly named household-population research domain is defensible from Census methodology. All group-quarters rows remain structurally present for ordinary analysis. This does not identify official CPS/SPM coverage or justify comparing an annual restricted rate directly with a published three-year full-universe state rate. Before declaring any unit INCLUDED/OUTSIDE, document source-year coverage, compatible ASEC restrictions, Armed Forces/residence treatment and unresolved coverage. Missing declarations remain unresolved.

Construction decisions to retain in provenance

• Usual hours describe the past year. Last-week hours and current employment status are different measurements. Preserve genuine observations, including 40; replace the materialized ACS defaults only through the authenticated source linkage.
• Age-15 development completion should use the complete public March 2025 age-15 donor cohort, whose source weights avoid inherited Build P support selection. The earlier 1,216 pooled/399 income-year-2024 retained records remain qualified historical and selected-support evidence. Full-source validation supports the simpler weighted empirical development choice. Both validation results and source allocations remain explicit.
• Younger-child zero completion is an explicitly selected modeling policy, not observed nonwork. All 510,098 younger ACS records have unknown raw earnings; available raw/mapped inputs show no earnings contradiction, but unknowns remain unknown in provenance.
• ACS does not observe a complete secondary-family graph. Accepted parent/spouse links precede residual rules. Adult nonrelative separation, unresolved-child reference pooling and constructed minor unit-reference roles are labeled assumptions. Minor unmarried-partner classification has a separate sensitivity. Contradictory accepted links fail; no fabricated parent pointer or adult is introduced to force a result.
• Group-quarters membership and ordinary population coverage remain intact; SPM measurement exclusions retain null outcomes rather than becoming nonpoor observations.
• County normalization preserves existing assigned location and makes valid codes five-digit strings. It does not turn PUMA-based assignment into an observed Census block.
• Reconstruct or regroup affected SPM-owned inputs explicitly. Never copy an old household-wide total into every new SPM unit. Leave unrelated tax-unit membership unchanged.
• Formula-owned SPM outcomes are calculated by the canonical software. Preserve authentic source observations separately; do not persist them as model overrides by accident.

The reconstruction approach follows the distinction between observed relationships and imputed pointers in Census's ACS SPM methodology, pp. 5–6. It is a versioned Microcosm construction, not a claim of exact 2024 Census unit replication.

Remaining release gates

• Qualify the complete public age-15 ASEC donor cohort, weights, work history and household-held-out comparison.
• Apply the reviewed empirical completion to only the authenticated eligible ACS cells and verify the full sidecar.
• Run the versioned ACS partition on complete small households, with reference/partner/minor/unrelated-family/GQ cases and explicit unit-input regrouping; actual 512-household monetary pilot passes.
• Pin and verify the actual country/Core/wrapper/calculator/Microdf combination, including native export/reload and supported geography filters.
• Produce a small repaired candidate and a full before/after input/membership inventory.
• Build the complete dense successor only after the pilot passes.
• Compare incumbent and repaired data on the same frozen target definitions, periods and tolerances; separate data changes from model changes using compatible old/new runtime cells.
• Check national/state/CD support, effective sample sizes, tails, population/fiscal fit, and protected target families. Do not copy the incumbent's passing verdict to changed calculations.
• Evaluate child/total/deep SPM poverty on matched universes and threshold vintages as held-out diagnostics, including partition/partner sensitivities. Do not calibrate to these holdouts to make the check pass; see #646.
• Run the actual SNAP counterfactual and input-distribution checks. Neither an unchanged result nor a merely nonzero result proves correctness.
• Perform a scoped recalibration if required, then rerun the affected comparisons without relaxing gates or hiding inherited limitations.
• Attach an immutable candidate manifest, exact compatibility evidence and dashboard tied to the accepted bytes; complete independent review before promotion.

There is no repaired dense candidate, native old/new nonregression result or release acceptance yet. Source PRs and synthetic tests do not certify a population. This tracker does not authorize a merge, default promotion or publication.

Related: #765, #32, #33, #646, #893.

Start by reviewing the remaining release gates and the linked work in #923–#926, PolicyEngine/spm-calculator#45, and policyengine-us#9462. The first concrete work is applying the reviewed age-15 completion only to authenticated ACS cells, then verifying the sidecar, country integration, runtime matrix, and national, state, and district comparisons. Done requires canonical SPM compatibility and no unexplained material deterioration against the pinned incumbent.