PolicyEngine / PolicyEngine/boe-var-model
Add numeric IRF-magnitude benchmarks against Brignone & Piffer (2025) Figures 2-3
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 13m
- Merged PRs (30d)
- 1
Description
Context
The calibration/validation suite is in good shape for what it covers:
tests/test_validation.pyhard-gates the Table 2 zero/sign restrictions on every accepted draw, FEVD share identities, historical-decomposition reconstruction, and the paper's headline FEVD benchmark (global shocks ~40% of UK GDP / ~50% of UK CPI variance) within a [30%, 60%] band on PRs, with the tighter importance-weighted magnitudes re-run nightly.tests/test_committed_artifacts.pygates the committed evaluation JSONs with 1e-3 tolerances plus a code-hash staleness guard.
The gap
Impulse-response magnitudes are validated only by sign and by the paper's unrestricted qualitative check (oil price rises after a world-supply shock). There is no tolerance-based comparison of median IRF paths / peak responses against the paper's Figures 2-3, because the paper (Brignone & Piffer, 2025, "A structural VAR model for the UK economy", docs/a-structural-var-model-for-the-uk-economy.pdf) publishes IRFs only as figures — no numeric table or replication CSV of IRF values exists in this repo or, as far as we can find, in any official published source.
Per project policy, reference numbers must come from authoritative sources, never invented or eyeballed from figure pixels.
What is needed
One of:
- Authors' replication package: numeric IRF medians and 68%/90% bands (per variable x shock x horizon 0-20) underlying Figures 2-3, e.g. by contacting the authors or checking any BoE Staff Working Paper replication release.
- A pinned in-repo reference: a one-off high-draw weighted run (same config as
test_weighted_fevd_matches_paper_benchmark) committed asresults/irf_reference.jsonwith acode_versionhash, then gated intest_committed_artifacts.pythe same way the evaluation artifacts are. This locks against regressions even without the paper's exact digits.
Option 2 is fully self-serve and consistent with the existing artifact-gating pattern; option 1 would additionally validate absolute agreement with the paper.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/test_validation.py and tests/test_committed_artifacts.py, then follow the test_weighted_fevd_matches_paper_benchmark configuration. Determine whether to obtain authoritative Figure 2–3 IRF values or create the pinned results/irf_reference.json artifact; done means numeric IRF checks are added with the existing code-hash and tolerance safeguards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100