aai-institute / aai-institute/setu
Enhancement: calibration diagnostics tailored to discrete estimators (reliability diagram / non-randomized PIT)
- Ngôn ngữ chính
- Python
- Star
- 4
- Fork
- 0
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
## Context
Likelihood-SBC ranks the true observation's `log_prob` among synthetic draws' `log_prob`s and tests the ranks for uniformity. For **discrete estimators** (`DiscreteNLE`, categorical/count data) the density value is tied across many synthetic draws that equal the true count. PR #7 fixes the immediate correctness problem by switching the rank to a **randomized PIT** (`#(<) + Uniform{0..#(==)}`), which is uniform under calibration for discrete predictive distributions (Czado, Gneiting & Held, 2009, *Predictive Model Assessment for Count Data*, Biometrics). Before that fix, the strict-`<` rank was systematically left-biased and made calibrated discrete estimators falsely fail SBC.
So rank-SBC is now **well-defined** for discrete data. This issue is about whether it is the **best** calibration diagnostic for discrete dimensions, and tracks nicer alternatives. Not a bug — the current behaviour is correct. **Post-talk follow-up.**
## Why revisit for discrete estimators
- The randomized PIT introduces mild RNG dependence (the KS statistic depends on the tie-breaking key).
- Ranking by `log_prob` value is a highest-density-style transform: valid, but blind to miscalibrations that rearrange mass while preserving the distribution of density values (a power limitation, not a definedness one).
- For discrete/count data there are more direct and interpretable calibration checks.
## Options to consider
1. **Total Variation on the PMF** — already implemented for discrete estimators in PR #7 (predicted vs. empirical PMF). Cleanest discrete goodness-of-fit; no randomization. Candidate to be the *primary* discrete calibration signal, with SBC as corroboration.
2. **Non-randomized PIT** (Czado et al. 2009) — deterministic PIT-reliability curve; removes the RNG dependence of the randomized rank and gives a reliability-diagram view.
3. **Reliability diagram / calibration curve** — for categorical predictions, binned predicted probability vs. observed frequency. Standard and interpretable for discrete/probabilistic-classification calibration.
## Suggested direction
- Keep the randomized-PIT rank fix (correct + standard) but document that discrete estimators lean primarily on TV, and
- add a non-randomized PIT / reliability-diagram check for discrete estimators as an optional, RNG-free calibration view.
## References
- Czado, Gneiting & Held (2009), *Predictive Model Assessment for Count Data*, Biometrics — randomized & non-randomized PIT.
- Talts, Betancourt, Simpson, Vehtari & Gelman (2018), *Validating Bayesian Inference Algorithms with Simulation-Based Calibration*.
Related: PR #7 (`add-discrete-nle`) — randomized tie-breaking in `setu/validation/_sbc.py`.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.