PolicyEngine / PolicyEngine/microcosm

WS-C deferred adjudications and the reconciliation layer

Open
#736 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
1d 3h
Merged PRs (30d)
94

Description

Central registry of the adjudications deferred during the #622 build-out (PR #735) and the calibration-target deferrals accumulated since, to be reviewed as the spine (#145/WS-E) and calibration workstreams converge on the first certified cut. Each item links its signed rationale in the committed resources (uk/target_reference_membership.json, uk/ledger_compile_parity_*_signed_differences.json, uk/calibration_measure_exclusions.json). Items graduate to their own issues when they become actionable. Spine-lane counterpart: #796.

Deferred adjudications from the #622 build-out

  • Property-income ×1.9 undercount → reconciliation home. The incumbent scales SPI Table 3.7 amounts ×1.9 (trace: policyengine-uk-data#311 / policyengine-uk-data#230 — SPI covers only taxpayers with liability; HMRC Property Rental Income Statistics show £46.68bn vs SPI ~£24.5bn for 2020-21). The family is signed-excluded from activation. Doctrine-clean paths: a chronicle package for the rental statistics + a declared reconciliation, or a declared value-scaling operation citing #230. This is the anchor case for the reconciliation layer.

  • Region-grain activation for the 18 two-level contract targets — Fixture A's 131 region×age production rows are structurally unmatched while activation pins country grain. Landed: microcosm#906 (merged 2026-09-15) fans the two-level targets over the region tier; the council-tax region controls are composed from the 296 authority rows since microcosm#934 (2026-09-16).

  • ons.population.scotland_households_3plus_children — the contract wants a household-composition fact that does not exist at ≤2025; needs a chronicle-side fact or an adjudicated alternative source.

  • TCL year surface — the incumbent pins the April-2025 publication values at {2026}; the Ledger side activates at 2025. Revisit when a 2026-vintage publication exists.

  • CGT 2025 level → #875 (2026-09-05). Superseded facts: since the Chronicle re-pin (microcosm#874) the latest outturn is 2024-25 (£127.3bn of gains, 584k taxpayers — the forestalling year ahead of the October 2024 rate rise) and the doctrine binds it at face value at the 2025 period; the engine's 2025-26 rates on those gains overshoot OBR's FY2025-26 receipts by +44.2% (obr.capital_gains_tax@2025). Ruled option 1: facts stay bound as published, the OBR miss is signed on the publication-stack register (obr.capital_gains_tax@2025, adjudication #875, window 2026-09-05 → 2026-10-05). Deliverable in #875: the index ruling for a declared 2024-25 → 2025 translation (rate-adjusted; a derived Chronicle fact), applying uprating_index in the compile step, the taxpayer count flat at 584k, and the cash-lag mapping for self-assessed OBR lines.

  • The 347 uprating holds — currently explicit identity holds; per-family index declarations belong to the materialization layer.

  • Salary-sacrifice canonical source — HMRC-table facts vs the incumbent's OBR-forecast rows (value ratios 0.81–1.14); pick per target when calibration binds them. (The five measure exclusions covering the family are windowed to 2026-11-25.)

  • UC payment-band source-only rows — 12 'No payment' / 'or over' facts skipped in fan-out (no safe incumbent inverse).

  • UC family-split month choice — resolved for the implemented paid-family/source-window comparison by #891/#892: explicit monthly windows and complete-month checks, including the twelve-month CY2025 paid headline/family/child margins, replace latest-month resolution. Remaining claim/cohort and administrative-family approximations stay in #882.

  • #623 disposition — resolved: the calibration seam landed via #729/#743, its gate story completed in PR #793, and #623 closes with that PR (residual acceptance work tracked here and in #796).

  • TCL retirement at the year move — the limit is abolished from 2026-04-06 (Budget 2025; policyengine-uk-data#465), so at period 2026 the 15 TCL targets are structurally zero: the references must retire or re-vintage with the calibration-year move (WS-F/#723). Add the contract-side guard from uk-data#465's step 2: a test refusing a reference whose binding variable is structurally zero at the reference period. (Our 2025 activation is unaffected — the limit is in force in FY 2025-26; when uk-data#465 moves the incumbent's values to 2025, Fixture B's 15 TCL ledger-only additions become matched rows at the next pin refresh.)

  • UC element-count targets (housing, LCWRA, carer, childcare — policyengine-uk-data#466, #738 step 3): no contract declarations exist; pipeline is chronicle facts first (childcare's published 160k series needs no API key — easiest first package; the rest need Stat-Xplore UC_Households exports), then a contract amendment per the #727 precedent, then references auto-activate on regeneration. Facts requested: PolicyEngine/chronicle#187.

  • Council tax level (microcosm#929 CT-C, graduated 2026-09-16 on closure): obr.council_tax (UK, net of CTR, £50.9bn) measures −12.8 % on the microcosm#934 candidate (−11.0 % on D2: the dwelling over-weighting that propped it up is gone). On the spine England's gross council tax averages about £1,522 per household against a requirement near £1,815 (MHCLG 2025-26 average Band D £2,280); the amount comes from frs_council_tax and is never tied to the authority's rate, and CTR is under-simulated (Scotland +17 %, Wales +9 % on the incumbent; uk-data#448). Reconciliation home: rules-based liability from band × the authority's Band D level (MHCLG Table 10 × 296, CTAS 2026 × 32 and the WG levels × 22 are all in the pinned feed) × the statutory ratios × the single-adult discount (CTB line 8 by band, also in the feed), with CTR engine-side (policyengine-uk#1669 / #150). Until then the OBR nation rows (obr.council_tax_{england,scotland,wales}, generator-pinned to E92/S92/W92 while Chronicle stamps K02000001) stay report-only, and a signed asymmetry on obr.council_tax for receipts on non-household dwellings (second homes and empties with premiums, which a household file cannot carry) is the honest disposition. The incumbent's housing/council_tax_net × 318 authorities stays blocked on its source (MHCLG taxbase × Band D proxy). Pairs with uk-data#448.

  • water_and_sewerage_charges raw-mapping divergence — resolved by fix-and-sign during the #686 lane: verified on the raw FRS 2024-25 tab (12,644 vs 14,307 nonzero households, all 1,663 differing households Scottish), the defect is the incumbent's NaN-propagating addition of the retired CSEWAMT header, and the candidate side stands. Signed as scottish-water-incumbent-nan-zeroing (defect_fix, +0.1009) on uk/spine_swap_signed_differences.json with a quantitative bound; latent in the incumbent since at least 2023-24 and surviving at 1.56.16.

  • Census household counts bind at the 2021/22 vintage in the first rowwise local candidate (revision 2026-09-02, microcosm#762 A11). The #802 bridge that reconciles the OA ladder's census household counts (650 constituencies + 361 local authorities) to the national household-composition partition cannot bind while three of its ten cells (multi_family_households, unrelated_adult_households, lone_parent_non_dependent_children_households) are #791's reviewed measure exclusions (in force to 2026-11-26). The rowwise candidate therefore classifies that bridge as unbound with a receipt (cross_grain.unbound_bridges in the candidate plan and manifest) and binds the 1,011 census household rows as published — 2021 for England, Wales and Northern Ireland, 2022 for Scotland — while every national row binds at the 2025 calibration period. The vintage gap is a known level pull (−2.58% household mass measured at K=2 in July); it is re-measured in the #762 receipts. Resolution rides #791 (the relationship-to-head column) or a shares/uprating ruling on the local family; until then the deferral is deliberate and visible, not silent. Revised 2026-09-03 (microcosm#762 A15): the shares/uprating ruling landed — every ladder household row now binds uprated to the calibration year by one national factor from the Ledger's ons.households_total fact (29,003,000 for 2025 over the ladder's 28,060,832 = 1.033577), receipted in the candidate manifest and mandatory under --release-candidate; the bridge stays unbound until #791. See the next item.

  • Five reviewed measure exclusions for national rows the spine cannot reach by reweighting (microcosm#762 A16, ruling 2026-09-03; window 2026-09-03 → 2026-10-03; erratum in the 2026-09-03 comment below). Register uk/calibration_measure_exclusions.json (49 entries): ons.savings_interest_income (spine £13.8bn vs ONS D.41 £95.5bn, HMRC SPI £18.3bn — a frame gap plus a concept gap) → #866; obr.housing_benefit (£2.7bn vs £12.1bn) → #867; slc.borrowers.plan_2_liable and slc.borrowers.plan_2_above_threshold (−65 % / −69 %) → #868; dwp.jsa_claimants (2.8×) → #869. The two ONS land rows first listed (8.7× / 2.5×) were withdrawn the same day: their misses were an artefact of per-block engine resolution in the candidate driver (receipt R15), and single-block resolution puts them within 3 % and 9 % of the ONS balance sheet. Each exclusion fails the run on expiry: correct the spine (the issues) or re-adjudicate. Also ruled the same day and applied on uk-rowwise-candidate-762-b: stretch bound 10, grain_equal, K=15 and 1500 epochs as UK local doctrine constants; A17, the census tenure cells uprate with the ladder rows.

  • Local target surface: 68 signed deferrals covering 513 cells (uk/local_target_reference_membership.jsonsigned_deferrals; no expiry — each retires when the fact lands or the frame supports it). Of 21,519 candidate references, 19,105 are active; 314 have no fact at or before the period, 1,586 no fact for the area, 513 are signed. By reason: msoa_mean_to_la_deferred (3 targets, 1,083 area-cells: ONS equivalised income is MSOA-grain means, LA aggregation waits on the mean-aggregation design — #147 lineage, no issue of its own); council_tax_voa_scotland_absent (256 cells: the VOA feed has no Scottish band counts — a chronicle-side fact ask, no issue yet); council_tax_wales_country_control_absent (176 cells, #762 A13: no Wales country stock-by-band fact — asked for in the 2026-09-03 comment, no issue yet); council_tax_ni_domestic_rates (88: NI has domestic rates, structurally absent); private_rent_pipr_* (4 reasons, 361 cells: the PIPR feed is 2026-06 only, after the period, missing four English authorities, Scottish BRMA grain, no NI — chronicle-side, no issue yet); uc_gb_only_ni_absent / uc_children_gb_only_ni_absent (101 cells: Stat-Xplore UC local facts cover GB only — no issue yet); spi_la_target_measure_coverage_absent / spi_pcon_self_employment_mean_absent (9 cells: SPI publishes medians only for E06000027, nothing for Scilly, one constituency lacks the SE mean — structurally absent); council_tax_city_of_london_band_a_suppressed (1: VOA suppression); local_authority_support_floor_excluded (43) and council_tax_band_h_spine_support_absent (296) are spine-side and tracked on #796. Revision 2026-09-16 (microcosm#934): 20,885 active, 341 signed cells; council_tax_voa_scotland_absent, council_tax_wales_country_control_absent, council_tax_ni_domestic_rates and council_tax_city_of_london_band_a_suppressed retired with dates under the census family (uk/uk_local_target_census.jsonretired_deferrals); the 297 band-H cells (296 English + Shetland) remain, tracked in #796.

  • Local binding adjudications (uk/local_binding_adjudications.json, 6 entries; #802 2026-08-31 → 2026-11-30, #762 A1 2026-09-02 → 2026-12-02): the fences lifted so the local families may bind — census disclosure-control noise; the SPI frame-model proxy; full-FRS TEI bands unavailable; population universe = private households; UC bound at benefit-unit grain; VOA dwellings vs the household frame. Each is a target-side reconciliation the family stands on; renewal or a real fix falls due at expiry. No issue yet. Revision 2026-09-16: voa_dwellings_vs_household_frame re-adjudicated onto the taxbase basis by microcosm#934 (approved 2026-09-15, expires 2026-12-15; England and Wales occupied chargeable ≈ households, Scotland chargeable +1.2 %).

  • Local uprating holds (17,077 cells; uprating_holds in the local membership): age 2024, SPI tax-year 2023 and the census tenure cells 2021/22 are held to 2025 as identity except tenure, which A17 uprates with the ladder rows; UC 2025-05 and council-tax stock 2025 carry none. Same resolution as the national 347 holds above: per-family index declarations in the materialization layer, or later facts (asked for in the 2026-09-03 comment).

The uk_target_fit disposition remainder (PR #793's adjudication queue)

The adjudicated exclusion pass (42 windowed entries, expiring 2026-11-26, correct-or-renew at apply time) landed in PR #793; the first certified cut blocks on what remains:

  • The 8 UC caseload/two-child-limit cells (−28.6%…−62.0% at the post-exclusion optimum): the pre-registered U8 take-up lever is measured and exhausted (0.85 leaves every cell unchanged while destroying housing benefit −0.0% → −52.4%); the measured lever is capital-test support — #750 / policyengine-uk#1824. Decide: wait for #750, retarget, or exclude.
  • hmrc/dividend_income_count_income_band_500_000_to_1_000_000 — the historical −36.7% failure recovered after #803; the post-fix validation records −2.2% on spine-i and −2.3% on spine-j. This closes that historical band-definition defect, not future release validation.
  • #792 — measure materialization is exclusion-set-dependent — resolved by #803 and the post-merge validation on #792. The cause was deterministic band-edge derivation from the pruned registry; the RNG hypothesis was refuted. Band edges now come from the compiled pre-exclusion register, with roster-invariance checks.
  • Doctrine posture for the certified cut: the Phase-3 run carries epochs 1500 / family_equal as receipted overrides (the 2026-08-24 ruling keeps uniform the default); the certification records them verbatim. Promote to doctrine, or certify with receipted overrides.
  • Fuel-duties ledger retarget (household-incidence fact vs total receipts — chronicle-side): the binding is verified correct; the gap is universe scope plus LCFS diary under-capture at 37% of implied litres. The target carries a windowed exclusion meanwhile.
  • The five publication-stack target-fit exclusions due 2026-09-30 — retired in #892: dwp.uc.households_single_with_children, dwp.uc.households_children_1, dwp.uc.households_children_2, dwp.uc.households_children_5_or_more, and hmrc/private_pension_income_count_income_band_100_000_to_150_000. The canonical comparison evidence records all five within the unchanged 25% bound at 1,500 updates / family_equal. Remaining UC diagnosis and remeasurement are consolidated in #882; this does not certify an artifact.
  • Sixth entry (2026-09-05, on uk-publication-stack-834, microcosm#874 composition): obr.capital_gains_tax@2025 (+44.2%, the 2024-25 forestalling-year gains bound at 2025 — #875), window 2026-09-05 → 2026-10-05. The #874 run-4 receipt also reports the private-pension band entry stale (+24.7%, back inside the bound; the rot rule wants it removed) and two unsigned state-pension SPI bands (50–70k +34.6%, 20–30k +25.3%) outside the bound.
  • Exclusion-window renewals fall due 2026-11-25/26: the 5 salary-sacrifice and 42 target-fit entries all fail closed on expiry — each needs its gap corrected or its adjudication renewed by then.

National calibration-target issues previously untracked here

  • #790 — adjudicate the incumbent's four post-calibration aggregate scalers (the rail/bus/fuel class deliberately left out of E6 scope)
  • #789 — UK bus fare and subsidy consumer targets from Chronicle facts
  • #738 — superseded by #882. Composition activation and the first candidate remeasurement landed; remaining element integration and remeasurement after the outstanding repairs are carried forward in #882.
  • #701 — superseded by #882. Missing UC caseload bindings are repaired; expenditure comparability and certified consumer acceptance are carried forward in #882.
  • #704 — sequence calibration after the amounts imputations; staging seam has no CGT target parity evidence
  • #731 — the scorecard (populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks) — becomes the acceptance probe of the first certified cut
  • #661 — publish Chronicle package IDs in calibration diagnostics targets

Refs: #622, #735, #665, #145, #796, PR #793.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unchecked adjudications and their signed resources, including uk/target_reference_membership.json, uk/ledger_compile_parity_*_signed_differences.json, and uk/calibration_measure_exclusions.json. Identify one item that has become actionable and follow its referenced issue or data source before proposing a separate change. Done is defined by the relevant item graduating to its own issue or receiving the stated contract, fact, test, or reconciliation decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.