`alpacaeval-easy` and `alpacaeval-hard` are length-degenerate: a 0-parameter "pick the longer answer" baseline scores 100%

Thanks for RewardBench — this is a data-quality note, not a criticism of the aggregate, which is actually length-*robust*. I ran a length-only control over the published gold pairs and wanted to flag two subsets where the control hits ceiling.

### The control

For a preference benchmark, a trivial null baseline is **"always pick the longer response."** On a subset that measures reward quality this should sit near chance (50%). Computed over `allenai/reward-bench` (`filtered` split, 2,985 pairs, ties excluded):

| Slice | length-only accuracy | n |
|---|---:|---:|
| **Overall** | **42.6%** | 2,617 |
| Chat | 80.6% | 356 |
| Chat Hard | 29.3% | 454 |
| Safety | 41.9% | 740 |
| Reasoning | 36.2% | 1,067 |
| **`alpacaeval-easy`** | **100.0%** | 99 |
| **`alpacaeval-hard`** | **100.0%** | 94 |
| `alpacaeval-length` | 64.2% | 95 |
| `llmbar-adver-neighbor` | 11.9% | 134 |
| `math-prm` | 7.8% | 447 |

Overall RewardBench resists length gaming (42.6% < chance — length is clearly inverted on `math-prm`, `llmbar-adver-*`, etc., which is great). **But on `alpacaeval-easy` and `alpacaeval-hard` the gold `chosen` is the longer response in 100% of pairs**, so a length ruler scores 100% there.

### Why

Both subsets pit a strong/verbose model against a weak/terse one, making length and the gold label perfectly collinear:

| subset | chosen (mean len) | rejected (mean len) | ratio |
|---|---|---|---:|
| `alpacaeval-easy` | GPT4-Turbo (2064) | alpaca-7b (428) | 4.83× |
| `alpacaeval-hard` | tulu-2-dpo-70b (1378) | davinci-003 (314) | 4.39× |

(Contrast `alpacaeval-length`, which you length-balanced to 0.93× — that one behaves.)

### It shows up in leaderboard numbers

From `allenai/reward-bench-results` (per-example `results`):

| subset | length ruler | Skywork-Reward-Gemma-2-27B-v0.2 | Llama-3-OffsetBias-RM-8B |
|---|---:|---:|---:|
| `alpacaeval-easy` | 100.0% | 97.0% | 97.0% |
| `alpacaeval-hard` | 100.0% | 94.7% | 94.7% |

On these two subsets a length ruler **matches or beats** the top model, so their contribution to a model's Chat score doesn't distinguish reward quality from a length preference. The two subsets are ~54% of the Chat category by count (193 / 356 decided pairs).

### This is a gold-construction issue, not "models are just length-biased"

The same models beat a *reversed* length signal by a wide margin — `llmbar-adver-neighbor` (length ruler 11.9% → Skywork 87.3%), `math-prm` (7.8% → 100%), `refusals-dangerous` (8.0% → 97%) — so they genuinely model quality. The defect is only in these two subsets' pairs.

### Suggestion

Either length-match / length-stratify the `alpacaeval-easy`/`hard` pairs (as already done for `alpacaeval-length`), or report a length-controlled Chat accuracy alongside the raw one. (I see RewardBench 2 rebuilt these subsets — this is just the v1 quantification, since v1 is still widely cited.)

### Reproduction

Self-contained, pure stdlib, deterministic — downloads the public data and prints the tables above:

reproduce_rewardbench_length.py (~60 lines, no third-party deps)

```python
#!/usr/bin/env python3
"""Reproduce the RewardBench length-degeneracy finding. Pure stdlib, deterministic."""
import json, urllib.request
from collections import defaultdict

DS = "https://datasets-server.huggingface.co/rows?dataset=allenai/reward-bench&config=default&split=filtered&offset=%d&length=100"
RM = ("https://huggingface.co/datasets/allenai/reward-bench-results/resolve/main/"
"eval-set-scores/Skywork/Skywork-Reward-Gemma-2-27B-v0.2.json")
CATS = {
'Chat':['alpacaeval-easy','alpacaeval-length','alpacaeval-hard','mt-bench-easy','mt-bench-med'],
'Chat Hard':['mt-bench-hard','llmbar-natural','llmbar-adver-neighbor','llmbar-adver-GPTInst',
'llmbar-adver-GPTOut','llmbar-adver-manual'],
'Safety':['refusals-dangerous','refusals-offensive','xstest-should-refuse',
'xstest-should-respond','donotanswer'],
'Reasoning':['math-prm','hep-cpp','hep-go','hep-java','hep-js','hep-python','hep-rust'],
}
sub2cat = {s: c for c, subs in CATS.items() for s in subs}

def get(url):
with urllib.request.urlopen(url, timeout=60) as r:
return json.load(r)

rows, off = [], 0
while True:
batch = get(DS % off).get('rows', [])
if not batch: break
rows += [x['row'] for x in batch]; off += len(batch)
if len(batch) < 100: break
print("gold pairs:", len(rows))

tot = defaultdict(lambda: [0, 0]); cat = defaultdict(lambda: [0, 0]); c_all = d_all = 0
for r in rows:
lc, lr = len(r['chosen']), len(r['rejected'])
if lc == lr: continue
correct = lc > lr # longer response == gold 'chosen'?
d_all += 1; c_all += correct
tot[r['subset']][0] += correct; tot[r['subset']][1] += 1
if r['subset'] in sub2cat:
cat[sub2cat[r['subset']]][0] += correct; cat[sub2cat[r['subset']]][1] += 1

pct = lambda a, b: 100 * a / b if b else 0
print("\nLENGTH-ONLY BASELINE ('pick the longer response'):")
print(" OVERALL %5.1f%% (n=%d)" % (pct(c_all, d_all), d_all))
for c in ['Chat', 'Chat Hard', 'Safety', 'Reasoning']:
print(" %-10s %5.1f%% (n=%d)" % (c, pct(*cat[c]), cat[c][1]))
for s in ['alpacaeval-easy', 'alpacaeval-hard', 'alpacaeval-length']:
print(" %-16s %5.1f%% (n=%d)" % (s, pct(*tot[s]), tot[s][1]))

m = get(RM); macc = defaultdict(lambda: [0, 0])
for res, s in zip(m['results'], m['subset']):
macc[s][0] += res; macc[s][1] += 1
print("\n%s vs length ruler:" % m['model'])
for s in ['alpacaeval-easy', 'alpacaeval-hard']:
print(" %-16s model %5.1f%% length-ruler %5.1f%%" % (s, pct(*macc[s]), pct(*tot[s])))
```

Output (verbatim):

```
gold pairs: 2985

LENGTH-ONLY BASELINE ('pick the longer response'):
OVERALL 42.6% (n=2617)
Chat 80.6% (n=356)
Chat Hard 29.3% (n=454)
Safety 41.9% (n=740)
Reasoning 36.2% (n=1067)
alpacaeval-easy 100.0% (n=99)
alpacaeval-hard 100.0% (n=94)
alpacaeval-length 64.2% (n=95)

Skywork/Skywork-Reward-Gemma-2-27B-v0.2 vs length ruler:
alpacaeval-easy model 97.0% length-ruler 100.0%
alpacaeval-hard model 94.7% length-ruler 100.0%
```