PolicyEngine / PolicyEngine/snap-qc-sim

Regime-robustness block for the distributional backtest

Open
#88 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
18m
Merged PRs (30d)
5

Description

Diane Schanzenbach's 8/27 feedback (email thread "sampling error"): the QC system made several discrete shifts across our FY2012–24 panel, which is why Bauer–Schanzenbach stayed post-pandemic. Answer with numbers, not argument.

Extend analysis/persistence_backtest.py (merged in #87) with a regime_robustness block:

  1. Regime-crossing test: fit variance components on FY2012–19 only; score FY2022–24 targets with the same CRPS/pinball/coverage battery. If the persistence widths transfer across the pandemic break, the "shifts did not break prediction" claim gets its strongest form.
  2. Post-pandemic-targets summary: the existing per-cell rows already cover targets 2022–24 — add a filtered summary table alongside the full-panel one.
  3. Memo + locks per the existing conventions (generated-memo equality, live input hashes, exact regeneration; see tests/test_persistence_backtest.py).

Context: the answers went to Diane in the held 8/27 reply ("models fit only on earlier years still score well on 2022-2024, and the persistence model wins 2024 outright") — this issue makes that claim a committed artifact. Existing per-year CRPS in persistence_backtest_results.json already shows 2022/2023/2024 results from expanding windows; the new piece is the frozen-2012–19 fit.

Gate: sol review before merge (both prior rounds' reports under ~/.cache/axiom-oracles/snap-fy27/backtest-review/).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read analysis/persistence_backtest.py and tests/test_persistence_backtest.py first, then inspect persistence_backtest_results.json and the existing memo, lock, input-hash, and regeneration conventions. Add the regime_robustness block with the frozen FY2012–19 fit, FY2022–24 scoring, and filtered summary; done means generated artifacts match exactly, live input hashes are locked, tests pass, and the sol review is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.