PolicyEngine / PolicyEngine/boe-var-model

Add numeric IRF-magnitude benchmarks against Brignone & Piffer (2025) Figures 2-3

Open
#11 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
1
Avg merge
13m
Merged PRs (30d)
1

Description

Context

The calibration/validation suite is in good shape for what it covers:

  • tests/test_validation.py hard-gates the Table 2 zero/sign restrictions on every accepted draw, FEVD share identities, historical-decomposition reconstruction, and the paper's headline FEVD benchmark (global shocks ~40% of UK GDP / ~50% of UK CPI variance) within a [30%, 60%] band on PRs, with the tighter importance-weighted magnitudes re-run nightly.
  • tests/test_committed_artifacts.py gates the committed evaluation JSONs with 1e-3 tolerances plus a code-hash staleness guard.

The gap

Impulse-response magnitudes are validated only by sign and by the paper's unrestricted qualitative check (oil price rises after a world-supply shock). There is no tolerance-based comparison of median IRF paths / peak responses against the paper's Figures 2-3, because the paper (Brignone & Piffer, 2025, "A structural VAR model for the UK economy", docs/a-structural-var-model-for-the-uk-economy.pdf) publishes IRFs only as figures — no numeric table or replication CSV of IRF values exists in this repo or, as far as we can find, in any official published source.

Per project policy, reference numbers must come from authoritative sources, never invented or eyeballed from figure pixels.

What is needed

One of:

  1. Authors' replication package: numeric IRF medians and 68%/90% bands (per variable x shock x horizon 0-20) underlying Figures 2-3, e.g. by contacting the authors or checking any BoE Staff Working Paper replication release.
  2. A pinned in-repo reference: a one-off high-draw weighted run (same config as test_weighted_fevd_matches_paper_benchmark) committed as results/irf_reference.json with a code_version hash, then gated in test_committed_artifacts.py the same way the evaluation artifacts are. This locks against regressions even without the paper's exact digits.

Option 2 is fully self-serve and consistent with the existing artifact-gating pattern; option 1 would additionally validate absolute agreement with the paper.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/test_validation.py and tests/test_committed_artifacts.py, then follow the test_weighted_fevd_matches_paper_benchmark configuration. Determine whether to obtain authoritative Figure 2–3 IRF values or create the pinned results/irf_reference.json artifact; done means numeric IRF checks are added with the existing code-hash and tolerance safeguards.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.