aai-institute / aai-institute/setu

Enhancement: calibration diagnostics tailored to discrete estimators (reliability diagram / non-randomized PIT)

Đang mở
#23 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
4
Fork
0
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

## Context

Likelihood-SBC ranks the true observation's `log_prob` among synthetic draws' `log_prob`s and tests the ranks for uniformity. For **discrete estimators** (`DiscreteNLE`, categorical/count data) the density value is tied across many synthetic draws that equal the true count. PR #7 fixes the immediate correctness problem by switching the rank to a **randomized PIT** (`#(<) + Uniform{0..#(==)}`), which is uniform under calibration for discrete predictive distributions (Czado, Gneiting & Held, 2009, *Predictive Model Assessment for Count Data*, Biometrics). Before that fix, the strict-`<` rank was systematically left-biased and made calibrated discrete estimators falsely fail SBC.

So rank-SBC is now **well-defined** for discrete data. This issue is about whether it is the **best** calibration diagnostic for discrete dimensions, and tracks nicer alternatives. Not a bug — the current behaviour is correct. **Post-talk follow-up.**

## Why revisit for discrete estimators

- The randomized PIT introduces mild RNG dependence (the KS statistic depends on the tie-breaking key).
- Ranking by `log_prob` value is a highest-density-style transform: valid, but blind to miscalibrations that rearrange mass while preserving the distribution of density values (a power limitation, not a definedness one).
- For discrete/count data there are more direct and interpretable calibration checks.

## Options to consider

1. **Total Variation on the PMF** — already implemented for discrete estimators in PR #7 (predicted vs. empirical PMF). Cleanest discrete goodness-of-fit; no randomization. Candidate to be the *primary* discrete calibration signal, with SBC as corroboration.
2. **Non-randomized PIT** (Czado et al. 2009) — deterministic PIT-reliability curve; removes the RNG dependence of the randomized rank and gives a reliability-diagram view.
3. **Reliability diagram / calibration curve** — for categorical predictions, binned predicted probability vs. observed frequency. Standard and interpretable for discrete/probabilistic-classification calibration.

## Suggested direction

- Keep the randomized-PIT rank fix (correct + standard) but document that discrete estimators lean primarily on TV, and
- add a non-randomized PIT / reliability-diagram check for discrete estimators as an optional, RNG-free calibration view.

## References
- Czado, Gneiting & Held (2009), *Predictive Model Assessment for Count Data*, Biometrics — randomized & non-randomized PIT.
- Talts, Betancourt, Simpson, Vehtari & Gelman (2018), *Validating Bayesian Inference Algorithms with Simulation-Based Calibration*.

Related: PR #7 (`add-discrete-nle`) — randomized tie-breaking in `setu/validation/_sbc.py`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.