PolicyEngine / PolicyEngine/microcosm
Checkpoint and cache identity omit the staged frame: reform-vector cache ignores seed, constraint matrix unfingerprinted, manual materializer bumps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Summary
Several build-identity keys omit inputs that change results, and the code already works around the gaps by hand. This is the input-side complement to #326 (which is specifically the missing solver_config_sha256 — not duplicated here) and #314 (crosswalk pinning, since fixed in the checkpoint key).
The gaps (each verified on main, 8264c49)
1. The target-frame checkpoint hashes the raw H5, not the staged frame it actually materializes from.
_target_frame_checkpoint_identity (tools/build_us_fiscal_refresh_release.py:1291-1315) keys on: base H5 SHA, pe-us version, seed, period, registry version, crosswalk SHA. But materialization runs on base_frame after immigration, take-up, hours, SNAP, eligibility, pregnancy, SCF wealth, ACA, and Medicaid stages have rewritten it. The code acknowledges this hole explicitly — tools/build_us_fiscal_refresh_release.py:161-167:
# 2: the medicaid_take_up stage (populace #331) changed base_frame's
# takes_up_medicaid_if_eligible before target-frame materialization, ...
# the checkpoint identity hashes the on-disk base dataset, not the staged frame,
# and would otherwise silently reuse pre-stage frames.
TARGET_FRAME_CHECKPOINT_MATERIALIZER_VERSION = 2
Every future stage edit must remember to bump this integer by hand; forgetting reuses a stale checkpoint silently.
2. The reform-vector cache excludes seed while the vectors are computed on a seed-dependent frame.
REFORM_VECTOR_CACHE_CONTEXT_KEYS (:155-160) is only {base sha, pe-us version, period, crosswalk}; commit/seed/registry are intentionally excluded (:1184-1187) so calibration-only reruns reuse the cache. But the per-household reform vectors are evaluated on the post-source-stage frame (:6124), and the take-up / SCF-wealth / pregnancy stages draw seeded Bernoullis — so two runs with the same base and row count but different --seed share cache entries computed under the other seed's frame, pairing stale reform vectors with a freshly computed baseline. The materializer-version constant guards the target-frame checkpoint but there is no equivalent key on the reform cache (only TARGET_MATERIALIZATION_CACHE_SCHEMA_VERSION).
3. The constraint matrix is not fingerprinted, and incumbent comparison keys on a hash that excludes it.
_target_surface_payload (packages/populace-calibrate/src/populace/calibrate/diagnostics.py:130-149) hashes target rows (names/periods/measure selectors/values); the matrix contributes only rows/columns/nnz. _assert_incumbent_target_surface_matches (:1822-1850) accepts incumbent diagnostics on target_surface.sha256 equality alone — an engine change that moves coefficients (A) while preserving the target rows (b) is treated as the same surface.
4. CLI-overridable donor inputs skip identity and provenance.
The default SCF extract is SHA-pinned in scf_wealth.py, but a --scf-summary-extract path is loaded directly (:5950-5961); its digest enters neither the checkpoint identity nor the build/release manifests. Same family: _download_base_h5 (:1596-1604) fetches the mutable root with no revision (the observed SHA is recorded after the fact, so the identity is sound, but the input is not re-fetchable at that identity), and the Ledger --ledger-facts-sha256 / --ledger-manifest-sha256 pins are optional with bare-JSONL feeds accepted (:593-617).
5. The exact selected support set is not propagated into release lineage.
SelectionReport.as_manifest() (packages/populace-build/src/populace/build/us_runtime/warm_start_selection.py:124-147) drops identities_sha256 (the selection-source manifest embeds and verifies it on load, but the release manifest's selection_source payload carries only mode/join key/counts/source provenance). The US country package (us/country_package.json) commits no selection resource. A release therefore does not record which records were selected in a directly comparable form.
Fix sketch
One canonical build-run identity (the umbrella #326 also feeds into):
- add a canonical post-source-stage frame hash and derive both the target-frame checkpoint key and the reform-vector cache key from it (this retires the manual materializer integer and closes the seed hole at once);
- include CSR
indptr/indices/data(or a content hash of them) plus stable record IDs intarget_surface, and require engine identity in incumbent comparisons; - record CLI-supplied donor/extract SHAs in the build manifest and require them in the checkpoint identity;
- carry
identities_sha256throughas_manifest()into the release manifest.
Acceptance criteria
- Two runs differing only in
--seedcannot share reform-vector cache entries. - Editing any source stage invalidates the target-frame checkpoint without a manual constant bump (post-source frame hash in the key).
-
target_surfacedistinguishes two calibrations whose targets match but whose constraint coefficients differ. - A CLI-supplied SCF extract's SHA-256 appears in the build manifest.
- The release manifest records the selection
identities_sha256when a selection source is used.
Cross-links
- #326 (solver-config identity — complementary, deliberately not duplicated here)
- #314 (crosswalk was the same class of gap; its fix pattern applies)
- #217 (checkpoint reuse across calibration-only runs — reuse safety depends on these keys)
- #346 (informed-init reselection will add more seed-dependent stages ahead of materialization)
- #377 (cross-register consistency — same lesson: local guards, no cross-artifact identity)
Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace _target_frame_checkpoint_identity and REFORM_VECTOR_CACHE_CONTEXT_KEYS in tools/build_us_fiscal_refresh_release.py, then inspect _target_surface_payload and incumbent matching in diagnostics.py. Review SelectionReport.as_manifest() and the CLI donor-input paths before defining one consistent identity across caches and manifests. Done means all five acceptance criteria pass, including distinct constraint fingerprints and propagated selection identities_sha256.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, data, release
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100