allenai / allenai/reward-bench
Multiple-comparison correction for per-subset comparisons (best-of-23 inflates "leads on subset X")
- Lingua principale
- Python
- Stelle
- 736
- Fork
- 99
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
### Summary
RewardBench scores each model across 23 subsets, and it's common to report the subset where a model leads. Because that's the best of 23 comparisons, the reported subset win is the maximum of many noisy numbers and is biased upward — so an uncorrected per-subset p-value overstates significance. A short note (or a small helper) applying a family-wise correction would help readers avoid over-reading subset-level differences.
### Why it matters — two real cases from the released results
Per-subset two-proportion test, then the best A-direction win corrected with Šidák, `1 − (1 − p)^23`:
| Case | Overall | Best-subset raw p | Šidák (K=23) | Read |
|---|---|---|---|---|
| Two ~tied models (0.5350 vs 0.5340) | tied | 0.009 | **0.19** | selection artifact — not significant |
| A is 0.4 pp *worse* overall | A worse | 0.00049 | **0.011** | genuine subset result, survives correction |
The pair is the point: the correction isn't a blanket "everything is noise" — one win survives, one doesn't, and which survives is independent of the overall ranking. Between two *identical* models at K=23 you expect ~0.6 "significant" subset wins by chance alone, so a single marginal subset win is the base rate, not a finding.
### Suggestion
- a one-paragraph caveat in the README/leaderboard docs about best-of-K selection across the 23 subsets, and/or
- a small helper that reports Šidák/Bonferroni-corrected per-subset comparisons.
Method is deterministic (per-subset two-proportion z-test, Šidák over the 23 subsets), and the numbers above are from the released results files. Happy to open a PR with the helper + a reproduction script if that'd be welcome.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start by locating the README or leaderboard documentation and the released results files referenced in the issue. Confirm how the 23 per-subset comparisons are produced, then make the agreed deliverable—a caveat or helper—reproduce the two stated cases and clearly report the corrected result.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- machine-learning
- Tipo di issue
- Funzionalità
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 55/100