PolicyEngine / PolicyEngine/microcosm

Make target materialization checkpoints reusable across calibration-only runs

Open
#217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Problem

Changing calibration settings such as epochs, loss weights, or optimizer options should not rerun expensive PolicyEngine target materialization. Today the fiscal refresh scorer/build path can rematerialize JCT reform vectors such as SALT even when the reform definition itself did not change.

This is especially visible when validating Populace on the full CD target surface: tools/score_us_fiscal_targets.py and tools/build_us_fiscal_refresh_release.py materialize JCT targets again because the cache identity includes broad build state such as build commit and target registry version. That is conservative, but too coarse for launch iteration.

Desired design

Use finer-grained checkpoints:

  • Reform-vector cache keyed by the inputs that actually determine per-household reform estimates: (base_h5_sha256, policyengine_us_version, reform_id, period, congressional_district_crosswalk_sha256 if geography-dependent). Do not include unrelated target registry changes or the whole Populace build commit when the reform code and PE-US version are unchanged.
  • Target-frame / constraint-matrix cache keyed by (base_h5_sha256, exact target_surface.sha256, materializer_code_version, policyengine_us_version).
  • Calibration-only reruns should load the cached target frame/matrix and only solve weights.

Acceptance criteria

  • A calibration-only rerun with the same base H5 and target surface does not rerun JCT/SALT reform simulations.
  • A target-surface semantic change still invalidates the target frame/matrix safely.
  • Two different H5s, such as incumbent vs candidate with different record counts, do not share incompatible per-household vectors.
  • Diagnostics/manifest record which checkpoint keys were used.

Context

This came up during the June 2026 full-CD Populace validation. A corrected target-surface score needed fresh materialization, but future changes to only epochs/loss/optimizer should not pay that cost again.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/score_us_fiscal_targets.py and tools/build_us_fiscal_refresh_release.py, tracing how target materialization and cache identity are currently handled. Verify the checkpoint keys against the desired inputs, then exercise calibration-only reruns and target-surface or H5 changes to confirm reuse and safe invalidation; diagnostics and manifests should record the keys used.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.