PolicyEngine / PolicyEngine/snap-qc-sim
Regime-robustness block for the distributional backtest
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 18m
- Merged PRs (30d)
- 5
Description
Diane Schanzenbach's 8/27 feedback (email thread "sampling error"): the QC system made several discrete shifts across our FY2012–24 panel, which is why Bauer–Schanzenbach stayed post-pandemic. Answer with numbers, not argument.
Extend analysis/persistence_backtest.py (merged in #87) with a regime_robustness block:
- Regime-crossing test: fit variance components on FY2012–19 only; score FY2022–24 targets with the same CRPS/pinball/coverage battery. If the persistence widths transfer across the pandemic break, the "shifts did not break prediction" claim gets its strongest form.
- Post-pandemic-targets summary: the existing per-cell rows already cover targets 2022–24 — add a filtered summary table alongside the full-panel one.
- Memo + locks per the existing conventions (generated-memo equality, live input hashes, exact regeneration; see
tests/test_persistence_backtest.py).
Context: the answers went to Diane in the held 8/27 reply ("models fit only on earlier years still score well on 2022-2024, and the persistence model wins 2024 outright") — this issue makes that claim a committed artifact. Existing per-year CRPS in persistence_backtest_results.json already shows 2022/2023/2024 results from expanding windows; the new piece is the frozen-2012–19 fit.
Gate: sol review before merge (both prior rounds' reports under ~/.cache/axiom-oracles/snap-fy27/backtest-review/).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read analysis/persistence_backtest.py and tests/test_persistence_backtest.py first, then inspect persistence_backtest_results.json and the existing memo, lock, input-hash, and regeneration conventions. Add the regime_robustness block with the frozen FY2012–19 fit, FY2022–24 scoring, and filtered summary; done means generated artifacts match exactly, live input hashes are locked, tests pass, and the sol review is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100