Entropy-calibration ideas from weighted Monte Carlo (Avellaneda et al. 2001): KL objective arm, ESS diagnostic, per-target tolerances, target sensitivities

Avellaneda, Buff, Friedman, Grandchamp, Kruk & Newman (2001), "Weighted Monte Carlo: a new technique for calibrating asset-pricing models," IJTAF 4(1) is finance independently deriving survey calibration: simulate an ensemble from a prior (their paths ↔ our records), then reweight it to hit linear benchmark constraints (their prices ↔ our targets) by minimizing KL divergence from the prior weights, solved in the dual (Boltzmann weights, L-BFGS over one multiplier per constraint). The core is equivalent to Deville–Särndal calibration estimation with the exponential distance, so most of it is familiar — but four things the finance treatment sharpened look directly useful for populace-calibrate:

1. A KL-to-anchor objective arm in the benchmark protocol. Our loss is capped relative error on targets (relative_error_loss) with L2 anchor regularization toward design/uniform weights (#309). The entropic alternative regularizes with KL to the anchor instead: closed-form exponential-tilt structure in the dual, strictly positive multiplicative weights, and the ESS interpretation below for free. Worth an arm in the frozen holdout protocol (same targets, same support, matched budgets): loss/within-10%, holdout MSRE, ESS, and weight-tail behavior vs the current objective — especially given the ratio-50 cap exists to fight exactly the tail behavior a KL penalty shapes natively.

2. Effective-sample-size diagnostic from relative entropy. Their §2: D(p‖u)/log ν measures support thinness — a weight vector concentrated on ν^α of ν records scores exactly 1−α. That's a principled, scale-free companion to the ESS numbers we already worry about (e.g. Build E's ESS 5,728 on a 337k support, populace#299) and belongs in calibration_diagnostics regardless of objective choice.

3. Per-target tolerances instead of exact-fit-or-cap. Their §4 minimizes χ²_w + KL, where each constraint gets a tolerance w_j and the residual scales like √w_j; exact fit is the w→0 limit, and the optimal multiplier is λ*_j = −(E_p[g_j] − C_j)/w_j. That's the right frame for targets of heterogeneous reliability — vintage-aged SOI levels vs current admin counts (the #212 residual class) — as a per-target reliability input rather than one global cap.

4. Target sensitivities as regression coefficients (§5). Under the entropic solution, ∂E_p[F]/∂C_k for any downstream estimate F is the multivariate regression coefficient of F's per-record contributions on the target loading matrix, computable from a single solve — no re-calibration per perturbed target. Concretely: "which calibration targets move this state poverty estimate," standard output next to per-target diagnostics, and an explanation tool when estimates shift after a target-surface update.

5. Variance reduction tells us which estimates calibration actually stabilizes (§6). Calibration is implicitly a control-variate scheme: a downstream estimate well spanned by the target loadings inherits large variance reduction (factor 1−R² against the target span); an orthogonal one inherits none. That's the missing precision theory for local-area estimates from the one-national-dataset design — an honest, per-estimate answer to "is this county number stabilized by the calibration or not."

Suggested first step: the benchmark arm (1) plus the ESS diagnostic (2), since both slot into existing machinery without touching the certified recipe; (3)–(5) are independent follow-ups.

Paper also on SSRN. Related survey-stats lineage: Deville & Särndal (1992); the entropy/exponential member of their distance family is this exact estimator.

🤖 Generated with Claude Code

Start with the existing benchmark protocol and calibration_diagnostics machinery, including relative_error_loss and the frozen holdout comparison described in the issue. Implement the KL objective benchmark arm and ESS diagnostic first; done means matched-budget results report target loss, within-10% performance, holdout MSRE, ESS, and weight-tail behavior without changing the certified recipe.