aai-institute / aai-institute/setu

Enhancement: calibration diagnostics tailored to discrete estimators (reliability diagram / non-randomized PIT)

Open
#23 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Context

Likelihood-SBC ranks the true observation's `log_prob` among synthetic draws' `log_prob`s and tests the ranks for uniformity. For **discrete estimators** (`DiscreteNLE`, categorical/count data) the density value is tied across many synthetic draws that equal the true count. PR #7 fixes the immediate correctness problem by switching the rank to a **randomized PIT** (`#(<) + Uniform{0..#(==)}`), which is uniform under calibration for discrete predictive distributions (Czado, Gneiting & Held, 2009, *Predictive Model Assessment for Count Data*, Biometrics). Before that fix, the strict-`<` rank was systematically left-biased and made calibrated discrete estimators falsely fail SBC.

So rank-SBC is now **well-defined** for discrete data. This issue is about whether it is the **best** calibration diagnostic for discrete dimensions, and tracks nicer alternatives. Not a bug — the current behaviour is correct. **Post-talk follow-up.**

## Why revisit for discrete estimators

- The randomized PIT introduces mild RNG dependence (the KS statistic depends on the tie-breaking key).
- Ranking by `log_prob` value is a highest-density-style transform: valid, but blind to miscalibrations that rearrange mass while preserving the distribution of density values (a power limitation, not a definedness one).
- For discrete/count data there are more direct and interpretable calibration checks.

## Options to consider

1. **Total Variation on the PMF** — already implemented for discrete estimators in PR #7 (predicted vs. empirical PMF). Cleanest discrete goodness-of-fit; no randomization. Candidate to be the *primary* discrete calibration signal, with SBC as corroboration.
2. **Non-randomized PIT** (Czado et al. 2009) — deterministic PIT-reliability curve; removes the RNG dependence of the randomized rank and gives a reliability-diagram view.
3. **Reliability diagram / calibration curve** — for categorical predictions, binned predicted probability vs. observed frequency. Standard and interpretable for discrete/probabilistic-classification calibration.

## Suggested direction

- Keep the randomized-PIT rank fix (correct + standard) but document that discrete estimators lean primarily on TV, and
- add a non-randomized PIT / reliability-diagram check for discrete estimators as an optional, RNG-free calibration view.

## References
- Czado, Gneiting & Held (2009), *Predictive Model Assessment for Count Data*, Biometrics — randomized & non-randomized PIT.
- Talts, Betancourt, Simpson, Vehtari & Gelman (2018), *Validating Bayesian Inference Algorithms with Simulation-Based Calibration*.

Related: PR #7 (`add-discrete-nle`) — randomized tie-breaking in `setu/validation/_sbc.py`.

Contributor guide

Open the contributing guide

Research direction

The issue references PR #7 and the file `setu/validation/_sbc.py`. Start by reviewing that PR and file to understand the current randomized PIT implementation for discrete estimators. Examine the existing total variation (TV) on PMF implementation. Research the non-randomized PIT and reliability diagram concepts from the cited papers. 'Done' would be implementing an optional, RNG-free calibration view for discrete estimators, likely as a new function or class in the validation module.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, numpy
Domain
ai, data, devtools, machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.