PolicyEngine / PolicyEngine/microcosm

Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256)

Open
#326 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Summary

No single cached/checkpointed artifact in the US release builder records the solver/calibration configuration that produced a given calibrated result. During the Build F campaign (populace#299) this did not cause a wrong result, but it nearly misled a diagnosis. Filing so the identity gap is closed before it bites a run where config genuinely changes the cached values.

What's already correct (so this issue is narrow)

Two things I initially suspected turned out to be fine, and I want to record that so nobody "fixes" them:

  • target_registry_version is already a content hash. TargetRegistry.version hashes the canonical spec JSON (every spec's name/value/source/sign/period). A feed edit that changes the target registry (e.g. v5→v6→v7 removing M-CHIP CHIP rows) therefore does invalidate the target-frame checkpoint. No feed-collision risk there.
  • The target-frame checkpoint is legitimately solver-independent. _target_frame_checkpoint_identity (keys: base_dataset_sha256, policyengine_us_version, seed, target_period, target_registry_version, congressional_district_vintage_crosswalk_sha256) caches pre-calibration PE-US engine outputs over the base pool. Those don't depend on λ / max_weight_ratio / epochs, so omitting solver config from this checkpoint's key is correct.

The real gap

The calibrated result (weights, loss, diagnostics, exported H5) depends on the full solver config — l2_lambda, l0_refit_lambda_share, refit_l2_lambda, max_weight_ratio, epochs, learning_rate, l2_anchor/refit_l2_anchor, warm-start identity, mass policy — but no artifact's identity records that config as a hash. The release manifest records the values in prose (default_dataset, options), but there is no solver_config_sha256 you can compare across runs to answer "did the calibration config change?" from identity alone.

Why it matters (Build F evidence, framed accurately)

The campaign ran back-to-back invocations differing only in --l2-lambda (0.08 vs default 0.0) and --max-weight-ratio (5 vs 50). An l2_lambda=0.08 run that produced +258% income tax immediately preceded a default-λ dense run. When the default-λ run then failed a downstream gate, checkpoint bleed was a natural first hypothesis.

It was NOT the cause — attempt 5 was run in a fresh --checkpoint-root (clean room), and its dense result was byte-identical on the flagged masses to the prior attempt despite the clean room, proving the behavior was deterministic pipeline logic, not stale cache. But had the clean-room control not been run, the identical numbers could have been misattributed to a stale checkpoint. A solver_config_sha256 on the calibrated artifacts (and any calibration-stage cache added later) would make that question answerable from identity, without a control run.

Proposed fix

  • Add solver_config_sha256 (canonical-JSON hash of the full calibration option set) to the identity/manifest of the calibrated artifacts and any future calibration-stage checkpoint whose values depend on solver config.
  • Add a test pinning the invariant that the target-frame checkpoint is solver- and (given target_registry_version) feed-independent, so that independence is a stated contract rather than an accident.

Context

Found during the Build F certified-lineage rerun (populace#299). Not a blocker for that campaign's conclusions.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating _target_frame_checkpoint_identity, the calibrated artifact identity or release manifest, and the calibration options that determine the result. Trace how checkpoint and manifest identities are built, then add the requested solver configuration hash and the test for target-frame solver and feed independence. Done means calibrated artifacts expose a comparable solver_config_sha256 and the independence invariant is pinned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
54/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.