Sweep λ_L2 in the production US build: ESS-vs-accuracy frontier for the sparse L0+refit dataset

Goal

Determine whether a positive L2 concentration penalty improves the effective sample size of the production sparse dataset without trading off target accuracy — or, failing that, publish the measured ESS-vs-accuracy frontier as evidence for keeping λ_L2 = 0. This executes the future-work item from the L0 paper ("trace effective-sample-size vs. accuracy trade-offs by varying λ_L2 and max_weight_ratio").

Blocked by #284 / PR #283 — without the refit-stage threading, λ_L2 never touches the shipped weights.

Baseline (paper probe, uncapped)

Step 0 — ground the baseline (~minutes)

Score the current production candidate's weights with the new effective_sample_size() before spending compute. The paper probe was uncapped, while the builder defaults to --max-weight-ratio 5.0 — the real production baseline may already be less concentrated than the table above.

Step 1 — primary grid, both stages (~6.5 h sequential)

Eight builds at λ_L2 ∈ {0, 1e-4, 3e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1} via --l2-lambda, everything else at production defaults (seed 0, 1,500 epochs, mass="conserve", cap 5.0, --l0-refit-lambda-share 0.8). Weight ratios are O(1), so the penalty term is O(λ) against a calibration loss of ~0.05 — the action should sit in 1e-3…1e-1; the outer points confirm the flat and over-regularized ends.

Mechanics:

• Reuse --target-frame-checkpoint + the target-materialization cache so each point costs only the ~49-min calibrate stage.
• Pass --skip-reform-validation --skip-demographics on grid points.
• Each run's calibration_diagnostics.json (schema v4) now carries final_loss, fraction_within_10pct, effective_sample_size, realized_max_weight_ratio, top_1pct_weight_share, plus selection-stage provenance — the frontier reads straight off the artifacts. The builder writes diagnostics even when release gates fail, so over-regularized points still yield data.

Step 2 — stage attribution (~3–4 h)

At the 2 most interesting λ values from step 1, isolate where the penalty does its work:

• selection-only: --l2-lambda x --refit-l2-lambda 0 — does penalizing which records get picked matter even with an unpenalized final fit?
• refit-only: --l2-lambda 0 --refit-l2-lambda x — spreading the shipped weights on a fixed support.

Control: selection-stage L2 shifts the retained count away from ~57k. Record n_selected_households per run; if drift exceeds ~10%, re-tune --l0-refit-lambda-share to restore the count so comparisons stay clean.

Step 3 — finalists

For the 1–2 frontier winners: full builds with reform validation (JCT in-sample + OBBBA out-of-sample backtests) and rotated-fold holdout analysis. A regularizer's payoff may show up as better out-of-sample behavior even at slightly higher in-sample loss.

Decision rule

Adopt the largest λ_L2 whose weighted loss stays within tolerance of the λ=0 baseline (e.g., no worse than the dense benchmark's 0.0507, vs the sparse baseline 0.0474) while ESS improves materially. Otherwise keep λ_L2 = 0 and record the frontier in the paper's next revision.

Notes

• Runs are manual CLI invocations of tools/build_us_fiscal_refresh_release.py on a machine with the build inputs (ledger facts artifact, base H5); there is no CI build pipeline.
• A small driver script looping the builder over the grid and tabulating the frontier from the diagnostics JSONs is worth adding alongside the sweep (tools/).

🤖 Generated with Claude Code

Start only after #284 / PR #283 lands, then use tools/build_us_fiscal_refresh_release.py and the target-frame checkpoint/cache to score the current production candidate with effective_sample_size(). Run the specified --l2-lambda grid and inspect each calibration_diagnostics.json, including ESS, loss, accuracy, and weight-ratio fields. Done means publishing the measured ESS-vs-accuracy frontier, stage-attribution results, and the decision on whether to retain or change λ_L2 = 0.