aai-institute / aai-institute/setu

Enhancement: calibration diagnostics tailored to discrete estimators (reliability diagram / non-randomized PIT)

Offen
#23 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
4
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

## Context

Likelihood-SBC ranks the true observation's `log_prob` among synthetic draws' `log_prob`s and tests the ranks for uniformity. For **discrete estimators** (`DiscreteNLE`, categorical/count data) the density value is tied across many synthetic draws that equal the true count. PR #7 fixes the immediate correctness problem by switching the rank to a **randomized PIT** (`#(<) + Uniform{0..#(==)}`), which is uniform under calibration for discrete predictive distributions (Czado, Gneiting & Held, 2009, *Predictive Model Assessment for Count Data*, Biometrics). Before that fix, the strict-`<` rank was systematically left-biased and made calibrated discrete estimators falsely fail SBC.

So rank-SBC is now **well-defined** for discrete data. This issue is about whether it is the **best** calibration diagnostic for discrete dimensions, and tracks nicer alternatives. Not a bug — the current behaviour is correct. **Post-talk follow-up.**

## Why revisit for discrete estimators

- The randomized PIT introduces mild RNG dependence (the KS statistic depends on the tie-breaking key).
- Ranking by `log_prob` value is a highest-density-style transform: valid, but blind to miscalibrations that rearrange mass while preserving the distribution of density values (a power limitation, not a definedness one).
- For discrete/count data there are more direct and interpretable calibration checks.

## Options to consider

1. **Total Variation on the PMF** — already implemented for discrete estimators in PR #7 (predicted vs. empirical PMF). Cleanest discrete goodness-of-fit; no randomization. Candidate to be the *primary* discrete calibration signal, with SBC as corroboration.
2. **Non-randomized PIT** (Czado et al. 2009) — deterministic PIT-reliability curve; removes the RNG dependence of the randomized rank and gives a reliability-diagram view.
3. **Reliability diagram / calibration curve** — for categorical predictions, binned predicted probability vs. observed frequency. Standard and interpretable for discrete/probabilistic-classification calibration.

## Suggested direction

- Keep the randomized-PIT rank fix (correct + standard) but document that discrete estimators lean primarily on TV, and
- add a non-randomized PIT / reliability-diagram check for discrete estimators as an optional, RNG-free calibration view.

## References
- Czado, Gneiting & Held (2009), *Predictive Model Assessment for Count Data*, Biometrics — randomized & non-randomized PIT.
- Talts, Betancourt, Simpson, Vehtari & Gelman (2018), *Validating Bayesian Inference Algorithms with Simulation-Based Calibration*.

Related: PR #7 (`add-discrete-nle`) — randomized tie-breaking in `setu/validation/_sbc.py`.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.