Per-category #1 rankings are within noise on their own samples (paired McNemar) — consider showing CIs / significance groups

Thanks for RewardBench — this is a constructive measurement note, not a bug report.

## Observation

The leaderboard displays a strict per-category order (Chat / Chat Hard / Safety / Reasoning). Because every model is scored on the **same** items, the correct #1-vs-#2 test is a **paired** McNemar on discordant items (an unpaired two-proportion test is conservative here). On the public per-example results, all four category #1-vs-#2 gaps fall inside the noise:

| Category | #1 vs #2 | discordant b/c | McNemar p |
|---|---|---|---|
| Chat | internlm2-7b vs -20b | 4/3 | 1.000 |
| Chat Hard | Skywork-Reward-Gemma-2-27B vs -Llama-3.1-8B | 30/23 | 0.410 |
| Safety | (same pair) | 29/27 | 0.894 |
| Reasoning | Skywork-Reward-Gemma-2-27B vs URM-LLaMa-3.1-8B | 39/26 | 0.137 |

(Category sample sizes: Chat n=358, Chat Hard n=456, Safety n=740, Reasoning n=1431.)

## This is specific, not blanket skepticism

The **aggregate** ranking has the sample size to discriminate: the overall #1 (Skywork-Reward-Gemma-2-27B) beats #2 at p=0.048 and #3 at p=0.008. It is only the per-category slots (n≈358–1431, near-ceiling accuracy) that their own samples cannot separate. So this isn't "the benchmark is bad" — the headline metric is fine; the per-category point-ranks are presented with more precision than their samples support.

## Suggestion

For the per-category columns, consider publishing confidence intervals or a significance-grouped ordering (letters/tiers) rather than a strict integer rank, so "#1 on Safety" isn't read as a measurable lead when #1-vs-#2 is a coin flip.

The same pattern partially recurs on **RewardBench 2** (`allenai/reward-bench-2-results`): 4 of 6 category #1s are tied with #2 (Focus p=0.50, Math p=1.00, Safety p=1.00, Ties p=0.50), while Factuality (p=0.016) and Precise IF (p<0.001) are real leads — so RB2 narrows but doesn't remove it.

## Reproduce

Pure stdlib, deterministic, no model inference. Downloads the public `allenai/reward-bench-results` per-example files, aligns the two models item-by-item within each category (asserting identical subset order), and runs paired McNemar:

```python
#!/usr/bin/env python3
"""
Reproduce: RewardBench per-category #1 rankings are not distinguishable from #2.

RewardBench's leaderboard displays a strict per-category order (Chat / Chat Hard / Safety /
Reasoning). Because every model is scored on the SAME items, the correct #1-vs-#2 test is the
PAIRED McNemar test on discordant items (an unpaired two-proportion test is conservative here).
This script downloads the public per-example results, aligns the two models item-by-item within
each category, and runs McNemar. Across all four categories the top gap is NOT significant.

No third-party deps, no model inference, no RNG. Deterministic. Pure stdlib.
"""
import json
import math
import urllib.request

BASE = ("https://huggingface.co/datasets/allenai/reward-bench-results/resolve/main/"
"eval-set-scores/")

CATS = {
'Chat': ['alpacaeval-easy', 'alpacaeval-length', 'alpacaeval-hard', 'mt-bench-easy', 'mt-bench-med'],
'Chat Hard': ['mt-bench-hard', 'llmbar-natural', 'llmbar-adver-neighbor', 'llmbar-adver-GPTInst',
'llmbar-adver-GPTOut', 'llmbar-adver-manual'],
'Safety': ['refusals-dangerous', 'refusals-offensive', 'xstest-should-refuse',
'xstest-should-respond', 'donotanswer'],
'Reasoning': ['math-prm', 'hep-cpp', 'hep-go', 'hep-java', 'hep-js', 'hep-python', 'hep-rust'],
}

# Each category's displayed #1 vs #2 among the top models (paths under BASE).
PAIRS = [
("Chat", "internlm/internlm2-7b-reward.json", "internlm/internlm2-20b-reward.json"),
("Chat Hard", "Skywork/Skywork-Reward-Gemma-2-27B-v0.2.json", "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2.json"),
("Safety", "Skywork/Skywork-Reward-Gemma-2-27B-v0.2.json", "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2.json"),
("Reasoning", "Skywork/Skywork-Reward-Gemma-2-27B-v0.2.json", "LxzGordon/URM-LLaMa-3.1-8B.json"),
]

_cache = {}

def load(path):
if path not in _cache:
with urllib.request.urlopen(BASE + path, timeout=120) as r:
d = json.load(r)
_cache[path] = (list(map(int, d["results"])), list(d["subset"]), d.get("model", path))
return _cache[path]

def chi_sf_1df(x):
"""Survival function of chi-square with 1 dof = erfc(sqrt(x/2))."""
return math.erfc(math.sqrt(x / 2.0))

def mcnemar(ca, cb):
b = sum(1 for x, y in zip(ca, cb) if x and not y)
c = sum(1 for x, y in zip(ca, cb) if y and not x)
stat = (abs(b - c) - 1) ** 2 / (b + c) if (b + c) > 0 else 0.0
return b, c, stat, chi_sf_1df(stat)

def main():
all_tied = True
print("Paired McNemar on each RewardBench category's displayed #1-vs-#2 gap:\n")
for cat, pa, pb in PAIRS:
ra, sa, na = load(pa)
rb, sb, nb = load(pb)
if sa != sb:
print(f"{cat}: subset order differs between files -> cannot index-align; SKIP")
continue
subs = set(CATS[cat])
ca = [x for x, s in zip(ra, sa) if s in subs]
cb = [y for y, s in zip(rb, sb) if s in subs]
b, c, stat, p = mcnemar(ca, cb)
tied = p >= 0.05
all_tied = all_tied and tied
print(f"{cat}: {na.split('/')[-1]} (acc={sum(ca)/len(ca):.3f}) vs "
f"{nb.split('/')[-1]} (acc={sum(cb)/len(cb):.3f})")
print(f" discordant b={b} c={c} | McNemar chi2={stat:.2f} p={p:.3f} -> "
f"{'TIED (rank not significant)' if tied else 'SIGNIFICANT'}\n")
# Contrast: the OVERALL #1 IS significant -- the board has power for its aggregate, just not per
# category. Item-level paired McNemar over the union of all four categories, #1 vs #2 overall.
all_subs = {s for ss in CATS.values() for s in ss}
g1 = "Skywork/Skywork-Reward-Gemma-2-27B-v0.2.json"
for label, g2 in [("#2 overall", "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2.json"),
("#3 overall", "LxzGordon/URM-LLaMa-3.1-8B.json")]:
ra, sa, na = load(g1)
rb, sb, nb = load(g2)
keep = [i for i, s in enumerate(sa) if s in all_subs]
ca = [ra[i] for i in keep]
cb = [rb[i] for i in keep]
b, c, stat, p = mcnemar(ca, cb)
print(f"OVERALL: {na.split('/')[-1]} vs {label} {nb.split('/')[-1]} -> "
f"McNemar p={p:.4f} {'(SIGNIFICANT: overall #1 is real)' if p < 0.05 else '(tied)'}")

print("=" * 70)
print("REPRODUCED: all four category #1 ranks statistically TIED with #2, "
"while the OVERALL #1 is significant"
if all_tied else "MISMATCH: at least one category rank was significant")
return 0 if all_tied else 1

if __name__ == "__main__":
import sys
sys.exit(main())
```

Prints `REPRODUCED: all four category #1 ranks statistically TIED with #2, while the OVERALL #1 is significant`.