Resolve publication audit findings before releasing the UK AI study

Summary

The study provides a useful comparison of how AI-related earnings losses affect households and public finances under different incidence and adjustment assumptions. Before publication, we should resolve the baseline and exposure-input validation gaps, correct the relative-poverty calculation and reporting inconsistencies, and complete the replication package.

This issue tracks the audit of commit 5d75106. It distinguishes confirmed corrections from unresolved validation questions and recommended extensions.

1. Validate the national baseline and exposure inputs

National baseline — unresolved validation question

The occupation table sums to approximately 22.95 million employees, consistent with the roughly 23.0 million implied by the 50-draw displacement mean and 7% target. ONS reports 29.63 million employees for January–March 2026. The enhanced-data sensitivity also raises the seed-0 central fiscal cost from £18.2bn to £29.5bn. These differences need reconciliation before the headline fiscal estimates are presented as national magnitudes.

• Publish a baseline validation table covering population, employees, employee earnings, income tax/NICs, benefit spending, poverty and Gini, with compatible official benchmarks.
• Explain differences in year, coverage, employment definitions, weighting and input-data versions, including the plain/enhanced discrepancy.
• Document the implications for the headline estimates and regenerate results where required.

This does not establish a weighting-code defect or justify automatically rescaling results. See the paper’s occupation totals and ONS benchmark, page 16.

Exposure inputs — unresolved provenance question

The derivation of the shipped C-AIOE series remains unresolved in the upstream crosswalk work. This does not establish that the rankings are wrong.

• Pin the upstream source/build commit and input versions.
• Document the order of standardisation, complementarity adjustment, crosswalking and employment aggregation.
• Rebuild and verify the packaged exposure CSVs from those inputs.
• Correct the methods’ claim that an additive complementarity offset is immaterial: theta / mean(theta) changes under an additive offset. Test or justify the retained level; do not silently subtract the reported offset.

2. Correct relative-poverty measurement

The pinned PolicyEngine UK implementation calculates the relative-poverty median using household weights, while the study reports person-weighted poverty rates. The median must also represent the person-level distribution of equivalised household income, consistent with DWP’s methodology.

A weighting diagnostic illustrates the difference: three equally weighted households with sizes 1, 1 and 5, and equivalised incomes £20k, £20k and £40k, produce household- and person-weighted medians of £20k and £40k respectively. At 60% of the median, the resulting person poverty rates are 0% and 28.57%. This demonstrates the weighting error; it does not quantify its effect on the study population.

• Use a consistent person-weighted median for relative-poverty calculations, including the Monte Carlo reporting path.
• Add an unequal-household-size regression test and reconcile the implementation with the backcast helper.
• Regenerate affected relative-poverty results and prose.

This finding concerns relative poverty. It does not establish an error in the absolute-poverty headline or Gini.

3. Align uncertainty labels, tables and figures

• Correct Figures 1 and 18: the code plots mean ± 1.96 × SD / sqrt(n), while captions describe intervals across assignments. Either label these as Monte Carlo confidence intervals for the mean or plot empirical assignment quantiles. Keep population and model uncertainty distinct. See Figure 1 calculation.
• Generate all Table 2 columns from the same draw convention. Displacement counts currently use seed 0, while outcome columns use 50-draw means. For example, the compression count is printed as 1.67m, versus a 50-draw mean of 1.605m. Clarify that displacement is fixed in expectation, with realised weighted totals varying. See table source.
• Reconcile Figures 8, 9 and 10 captions with their actual panels, signs and plotted series.

4. Complete reproducibility and integration checks

• Extend the manifest checker to validate declared code and presentation hashes, alongside inputs/results. Cover packaged crosswalks and the lockfile, and attest the final submission build. The current checker ignores code/presentation sections, and both presentation hashes differ from their recorded values.
• Document how an authorised replicator obtains every required input. The downloader omits the enhanced FRS file, constituency weights, constituency CSV and boundary GeoJSON required by the full build.
• Provide a locked environment recipe, immutable data revisions/checksums, and explicit pytest and LaTeX/latexmk prerequisites.
• Add synthetic integration checks against the pinned model for employee displacement, UC households, dual employment/self-employment, and final income/poverty statistics. Assert final tax-benefit outcomes, not only shock arithmetic.
• Complete and archive an end-to-end rebuild of the results and paper.

The hash discrepancies establish a provenance problem; the generated numerical macros checked during the audit matched the saved results.

5. Reconcile the paper’s claims and methods

• Narrow the abstract/introduction’s inequality claim to the tested pure job-loss families and grid. The mixed-adjustment table includes 0.42m displaced workers with a −0.03pp Gini change.
• Name the abstract’s poverty comparator: the matched wage-cut case raises poverty by 0.10pp relative to baseline, compared with 1.81pp under job loss.
• Reconcile the obsolete 7% wage-bill description with the matched 7.77% seed-0 loss, distinguish seed-0 policy illustrations from 50-draw summaries, and clarify self-employment versus capital-income coverage.
• State the pinned model’s absolute-poverty reference year and uprating accurately, and identify the vintage/definition of official comparisons.
• Remove the unsupported claim that stronger youth incidence necessarily raises inequality and lowers fiscal cost.
• Present the recession comparison as a historical-scale illustration unless it is rebuilt using compatible historical data/rules. Remove attribution of the residual to specific omitted mechanisms and unsupported upper-bound language; reconcile unemployment-rate and employee-displacement denominators.
• Correct the fiscal explanation of wage cuts versus displacement: benefits cushion household losses and increase fiscal cost. Support the explanation with tax/benefit components.

Recommended sensitivities and editorial improvements

These are extensions to prioritise, rather than confirmed errors requiring a particular replacement model:

• An internally coherent unemployment-duration or hiring scenario; retain a clear description of the current half-earnings/zero-hours hybrid.
• Alternative exposure-to-risk mappings, including a positive floor or rank-based mapping, and finer occupation imputation.
• Capital-return and dividend-only sensitivities around the inherited 39.8% interest/dividend uplift.
• More descriptive scenario labels and a shorter narrative, preserving necessary qualifications while moving implementation history into replication documentation.

Validation already performed

• 50 pure-function tests passed. No existing tests exercised final tax-benefit calculations.
• 111 numerical macros, 106 result hashes and 34 script hashes matched.
• The 44-page PDF was text-checked; 16 pages were visually inspected without clipping or unresolved references observed.
• Full population reproduction and exact-version model integration remain unverified because the licensed inputs were unavailable.

Completion criteria

Close this issue when the confirmed corrections are implemented, the two input-validation questions have documented resolutions, affected results and prose have been regenerated, and the final replication package passes its tests and complete manifest check. Record decisions on the recommended extensions separately.

Start with analysis/monte_carlo_families.py, analysis/replicate_jr16.py, paper/sections/results.tex, and analysis/rebuild.py, then run the existing tests and manifest checks. Resolve the confirmed calculation, reporting, provenance, and reproducibility findings, document the two validation questions, regenerate affected outputs, and complete an end-to-end rebuild that satisfies the stated completion criteria.