PolicyEngine / PolicyEngine/microcosm

Canonical multispine operator ordering: assemble spines first, then clone/impute/derive/seed/simulate/calibrate

Open
#395 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

The canonical operator ordering on the multispine

Max's architecture directive (2026-07-11): build stages are operators on ONE assembled multispine frame, in this order — not per-source pipelines that each re-implement enrichment:

  1. Harmonize (per-source, the only per-source layer): map each survey's raw observations into shared harmonized columns with universe metadata (e.g. the six-question disability battery from both ASEC PEDIS and ACS DEAR/DEYE/DREM/DPHY/DDRS/DOUT).
  2. Assemble the multispine: all survey spines (ASEC, ACS 1-yr, future: ACS 5-yr, SIPP) into one frame with spine tags and mass allocation.
  3. PUF cloning as a frame operator: fused tax-detail twins for records on EVERY spine, conditioned on the harmonized demographic+income surface — the ACS spine gets direct fusion instead of QRF transfer from the ASEC donor (retires the two-hop imputation for PUF-origin detail).
  4. Cross-source imputations (SCF wealth, SIPP tips, MEPS, …) once, on the pooled frame.
  5. Deterministic derivations (is_disabled via the SSI alignment rule over the harmonized battery, structural own_children_in_household, …) once, spine-agnostically.
  6. Runtime seeds (take-up draws) — already frame-level today.
  7. Engine materialization / microsimulation for target measures.
  8. Calibration, then selection/export, with gates at each stage boundary.

Where today deviates

The ASEC×PUF donor pool is built first (steps 3–4 run on the ASEC spine only), and the ACS spine attaches afterward via QRF transfer from that donor (acs_transfer.py). This is the shipped v1 (launch) architecture; it is honest and provenance-tracked, but PUF-origin detail on ACS carries two modeling hops, and per-source derivation risks rule drift (being fixed frame-level for disability/children on the buildl-acs-spine branch).

Scope of this issue

  • Migrate PUF cloning from the ASEC base-pool build to a multispine frame operator (design: conditioning surface, clone mass semantics per spine, memory plan — ACS 1-yr doubles to ~2.6M rows).
  • Migrate SCF/SIPP/MEPS imputation stages to run post-assembly on the pooled frame.
  • Certified-path guard throughout: each migrated stage must prove bit-identical output on the ASEC spine vs the current pipeline before the per-source implementation is retired (same guard pattern as the disability-derivation refactor).
  • Sequencing: post-launch; the v1 transfer architecture ships Monday with its provenance register and deferred-inputs list documenting exactly what the reorder improves.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with acs_transfer.py and the current ASEC×PUF donor-pool build to map where assembly and post-assembly stages currently occur. Use the disability-derivation guard pattern as the reference for proving bit-identical ASEC output, then verify pooled-frame ordering, per-spine clone mass semantics, and stage-boundary gates before retiring per-source implementations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.