`Focus` subset is length-confounded: a content-blind "pick the shortest completion" baseline scores 52.7% (2.1× chance, z=14.2)

Thanks for RewardBench 2 — first, credit where due: **the aggregate is length-robust.** A content-blind "pick the longest completion" ruler scores **21.3% overall, below the 24.8% chance rate**, so v2 clearly fixed the v1 issue where a length ruler hit 100% on the alpacaeval subsets (#267). This is a note about one subset where a length ruler still beats chance by a wide margin — in the *opposite* direction.

### The control

For a best-of-N reward benchmark (each item = 1 `chosen` + several `rejected`), a trivial null is **"pick the completion with the extreme length"** and check how often it's the gold `chosen`. On a subset that measures reward *quality* this should sit at chance (1 / n_completions ≈ 0.25 here). Computed over all 1,865 items of `allenai/reward-bench-2`:

| subset | n | chance | pick-LONGEST | **pick-SHORTEST** | mean len chosen | mean len rejected | z (shortest) |
|---|---:|---:|---:|---:|---:|---:|---:|
| **Focus** | 495 | 0.250 | 0.125 | **0.527** | 2216 | 2980 | **14.2** |
| Math | 183 | 0.250 | 0.224 | 0.377 | 1155 | 1337 | 4.0 |
| Safety | 450 | 0.250 | 0.247 | 0.311 | 1048 | 1313 | 3.0 |
| Factuality | 475 | 0.250 | 0.289 | 0.283 | 1684 | 1621 | 1.7 |
| Ties | 102 | 0.221 | 0.000 | 0.270 | 10 | 49 | 1.2 |
| Precise IF | 160 | 0.250 | 0.281 | 0.250 | 1686 | 1657 | 0.0 |
| **OVERALL** | 1865 | 0.248 | **0.213** | — | — | — | — |

On **Focus** (≈27% of the benchmark), the gold `chosen` is on average **shorter** than the rejected completions (2216 vs 2980 chars), and a zero-parameter "pick the shortest completion" baseline scores **52.7%** — more than double chance (binomial z = 14.2, p ≪ 1e-6). It holds under token count too (0.511). Math and Safety show weaker versions of the same inverse skew.

### Why it happens

The Focus rejected completions are generated to be "off-topic / unresponsive" (Section 3, following LLMBar). Empirically those off-topic completions tend to be **longer** (they ramble / pad), so length ends up partially collinear with the label — just inverted relative to v1's alpacaeval subsets.

### Why it's worth flagging

A model's Focus score is then confounded with its **length preference**, not only its focus-quality:

- A reward model with a **brevity** bias gets a Focus tailwind for the wrong reason (a length ruler already recovers >half the subset).
- The far more common **verbosity**-biased reward model is *penalized* on Focus for producing/preferring longer text, not for failing to detect off-topic answers.

Either way, Focus (and to a lesser extent Math/Safety) doesn't cleanly separate "prefers on-topic answers" from "prefers shorter answers."

### Honest caveats (this is a confound, not proof of a broken subset)

- Correct-is-shorter may be **partly legitimate**: a focused answer genuinely tends to be more concise, so length here is a *proxy* for focus, not purely spurious. I can't fully disentangle proxy-vs-artifact from the data alone.
- This is **not** "reward models are just length-biased." The same content-blind ruler is *below* chance overall and on several subsets — the models do model quality. The skew is localized to how these subsets' completions were generated.

### Suggestion

Report a **length-controlled Focus accuracy** alongside the raw one (e.g. length-stratify or length-match chosen vs rejected within Focus), or publish the per-subset length-only baseline above so readers can see which subsets are length-separable. This is the same fix AlpacaEval adopted (length-controlled win-rate) after its length confound was quantified.

### Reproduction

Self-contained, pure stdlib, deterministic — downloads the public dataset and prints the table above:

reproduce_rewardbench2_focus_length.py (~70 lines, no third-party deps)

```python
#!/usr/bin/env python3
"""Reproduce the RewardBench 2 per-subset length-confound finding. Pure stdlib, deterministic."""
import json, urllib.request, math
from collections import defaultdict

DS = "https://datasets-server.huggingface.co/rows?dataset=allenai/reward-bench-2&config=default&split=test&offset=%d&length=100"

def get(url):
for attempt in range(4):
try:
with urllib.request.urlopen(url, timeout=60) as r:
return json.load(r)
except Exception:
if attempt == 3:
raise

rows, off = [], 0
while off < 1865:
batch = get(DS % off).get("rows", [])
if not batch:
break
rows += [x["row"] for x in batch]; off += len(batch)
print("items:", len(rows))

def pick_by_len(comps, longest):
key = max(len(t) for t, _ in comps) if longest else min(len(t) for t, _ in comps)
tied = [lab for t, lab in comps if len(t) == key]
return sum(tied) / len(tied) # split ties fractionally (deterministic)

stat = defaultdict(lambda: {"n": 0, "short": 0.0, "long": 0.0, "chance": 0.0, "clen": 0.0, "rlen": 0.0})
for r in rows:
ch = r["chosen"] if isinstance(r["chosen"], list) else [r["chosen"]]
rej = r["rejected"] if isinstance(r["rejected"], list) else [r["rejected"]]
comps = [(t, 1) for t in ch] + [(t, 0) for t in rej]
if len(comps) < 2:
continue
s = stat[r.get("subset", "?")]; s["n"] += 1
s["short"] += pick_by_len(comps, longest=False)
s["long"] += pick_by_len(comps, longest=True)
s["chance"] += len(ch) / len(comps)
s["clen"] += sum(len(t) for t in ch) / len(ch)
s["rlen"] += sum(len(t) for t in rej) / len(rej)

z = lambda acc, ch, n: (acc - ch) / math.sqrt(ch * (1 - ch) / n) if n else 0.0
tot = {"n": 0, "long": 0.0, "chance": 0.0}
print("\n%-14s %5s %8s %10s %10s %8s %8s %8s" % ("subset","n","chance","PICK-LONG","PICK-SHORT","clen","rlen","z(short)"))
for s, v in sorted(stat.items(), key=lambda x: -x[1]["short"] / max(1, x[1]["n"])):
n = v["n"]; ch, sh, lo = v["chance"]/n, v["short"]/n, v["long"]/n
print("%-14s %5d %7.3f %9.3f %10.3f %8.0f %8.0f %8.1f" % (s, n, ch, lo, sh, v["clen"]/n, v["rlen"]/n, z(sh, ch, n)))
tot["n"] += n; tot["long"] += v["long"]; tot["chance"] += v["chance"]
print("\nOVERALL n=%d chance=%.3f PICK-LONG acc=%.3f (aggregate is length-robust)" % (tot["n"], tot["chance"]/tot["n"], tot["long"]/tot["n"]))
```

Output (verbatim):

```
items: 1865

subset n chance PICK-LONG PICK-SHORT clen rlen z(short)
Focus 495 0.250 0.125 0.527 2216 2980 14.2
Math 183 0.250 0.224 0.377 1155 1337 4.0
Safety 450 0.250 0.247 0.311 1048 1313 3.0
Factuality 475 0.250 0.289 0.283 1684 1621 1.7
Ties 102 0.221 0.000 0.270 10 49 1.2
Precise IF 160 0.250 0.281 0.250 1686 1657 0.0

OVERALL n=1865 chance=0.248 PICK-LONG acc=0.213 (aggregate is length-robust)
```