PolicyEngine / PolicyEngine/microcosm
Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 94
Description
Summary
No single cached/checkpointed artifact in the US release builder records the solver/calibration configuration that produced a given calibrated result. During the Build F campaign (populace#299) this did not cause a wrong result, but it nearly misled a diagnosis. Filing so the identity gap is closed before it bites a run where config genuinely changes the cached values.
What's already correct (so this issue is narrow)
Two things I initially suspected turned out to be fine, and I want to record that so nobody "fixes" them:
target_registry_versionis already a content hash.TargetRegistry.versionhashes the canonical spec JSON (every spec's name/value/source/sign/period). A feed edit that changes the target registry (e.g. v5→v6→v7 removing M-CHIP CHIP rows) therefore does invalidate the target-frame checkpoint. No feed-collision risk there.- The target-frame checkpoint is legitimately solver-independent.
_target_frame_checkpoint_identity(keys:base_dataset_sha256,policyengine_us_version,seed,target_period,target_registry_version,congressional_district_vintage_crosswalk_sha256) caches pre-calibration PE-US engine outputs over the base pool. Those don't depend on λ / max_weight_ratio / epochs, so omitting solver config from this checkpoint's key is correct.
The real gap
The calibrated result (weights, loss, diagnostics, exported H5) depends on the full solver config — l2_lambda, l0_refit_lambda_share, refit_l2_lambda, max_weight_ratio, epochs, learning_rate, l2_anchor/refit_l2_anchor, warm-start identity, mass policy — but no artifact's identity records that config as a hash. The release manifest records the values in prose (default_dataset, options), but there is no solver_config_sha256 you can compare across runs to answer "did the calibration config change?" from identity alone.
Why it matters (Build F evidence, framed accurately)
The campaign ran back-to-back invocations differing only in --l2-lambda (0.08 vs default 0.0) and --max-weight-ratio (5 vs 50). An l2_lambda=0.08 run that produced +258% income tax immediately preceded a default-λ dense run. When the default-λ run then failed a downstream gate, checkpoint bleed was a natural first hypothesis.
It was NOT the cause — attempt 5 was run in a fresh --checkpoint-root (clean room), and its dense result was byte-identical on the flagged masses to the prior attempt despite the clean room, proving the behavior was deterministic pipeline logic, not stale cache. But had the clean-room control not been run, the identical numbers could have been misattributed to a stale checkpoint. A solver_config_sha256 on the calibrated artifacts (and any calibration-stage cache added later) would make that question answerable from identity, without a control run.
Proposed fix
- Add
solver_config_sha256(canonical-JSON hash of the full calibration option set) to the identity/manifest of the calibrated artifacts and any future calibration-stage checkpoint whose values depend on solver config. - Add a test pinning the invariant that the target-frame checkpoint is solver- and (given
target_registry_version) feed-independent, so that independence is a stated contract rather than an accident.
Context
Found during the Build F certified-lineage rerun (populace#299). Not a blocker for that campaign's conclusions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating _target_frame_checkpoint_identity, the calibrated artifact identity or release manifest, and the calibration options that determine the result. Trace how checkpoint and manifest identities are built, then add the requested solver configuration hash and the test for target-frame solver and feed independence. Done means calibrated artifacts expose a comparable solver_config_sha256 and the independence invariant is pinned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 54/100