PolicyEngine / PolicyEngine/firm-microsim-paper

Restore VAT-liability-by-sector as a calibration target

Open
#2 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
Avg merge
5h 5m
Merged PRs (30d)
10

Description

Background

vat_liability_sector (HMRC net VAT liability by trade sector) has been removed from the calibration objective — see Config.calibrate_vat_liability_sector = False (gated in firm_microsim/calibration.py). It is still computed and reported as an informational diagnostic in firm_microsim/validate.py, but it no longer feeds the weight optimizer.

Why it was removed

The model fixes firm inputs and sets liability = turnover − input, but does not yet calibrate the input/output tax structure. As a result the sector-level net liability is structurally unhittable — it over/under-shoots for individual sectors (badly for small ones) and competed with the dimensions the paper relies on. Concretely it scored 43.9% (2023-24) and −121.1% (2024-25, sign-broken), dragging the naive 6-dimension mean down even though the five core dimensions calibrate well (~89–90%).

Removing it lets the optimizer focus on the five calibrated dimensions (HMRC turnover bands, ONS population, employment, sector counts, VAT liability by band) and gives a cleaner, defensible headline accuracy.

What restoring it requires

Restoration should come after the input/output calibration work (tracked in #1), specifically:

  • Calibrate the input/output tax structure using HMRC T9 Input and output tax and/or ONS ABS intermediate consumption, so per-sector net liability is reproducible rather than a by-product of unconstrained input draws.
  • Then re-enable the target by setting Config.calibrate_vat_liability_sector = True and (likely) raising vat_liability_sector_importance.
  • Consider flooring the per-sector accuracy metric at 0 (the current sign-aware metric is uncapped and over-penalizes tiny sectors), so the dimension is scored fairly once calibrated.

Acceptance

  • With input/output calibration in place, vat_liability_sector accuracy is positive and reasonable (target ≥ ~70%).
  • Re-enabling it does not materially degrade the five core dimensions.

Related: #1 (broader future calibration extensions).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading issue #1 and the calibration gate in firm_microsim/calibration.py, then inspect the vat_liability_sector diagnostic in firm_microsim/validate.py. Calibrate the input/output tax structure using the cited HMRC or ONS data before re-enabling Config.calibrate_vat_liability_sector. Done means sector accuracy is positive and about 70% or higher without materially reducing the five core dimensions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.