allenai / allenai/reward-bench

eval-set-scores: first ten Factuality ids are wrong in 179 of 188 files; scoring protocol is not recorded

Aperta
#273 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
736
Fork
99
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Two things about `allenai/reward-bench-2-results` that we hit while joining our own runs
against the published per-item scores.

### 1. Wrong ids in the first ten Factuality entries

`eval-set-scores/.json` stores parallel arrays, so `(subset, id)` is the only way to
line an item up against anything else. In **179 of 188 files** the first ten Factuality
entries are labelled `30`–`39` instead of `0`–`9`, producing ten duplicate keys per file.

```
first ten Factuality ids, affected : ['30','31','32','33','34','35','36','37','38','39']
expected : ['0','1','2','3','4','5','6','7','8','9']
```

All 179 affected files show the identical shape. The 9 unaffected ones are the seven
Skywork-Reward-V2 models, `Skywork/Skywork-VL-Reward-7B`, and `HFXM/RAMO-Llama3.1-8B`.

Subset totals are unaffected: non-Ties scoring takes a mean and never reads the id, and the
Ties subset, which does read it, is not among the affected entries. What breaks is joining
on id — ten items attach to the wrong row and nothing raises.

We ran into this before we went looking for it. Validating our harness against
`LxzGordon/URM-LLaMa-3.1-8B`, our per-item join covered 1753 of 1763 items rather than all
of them, and within Factuality 465 of 475 — ten short in both cases, matching the ten
duplicate keys. The two Skywork-Reward-V2 models we validated alongside it joined
completely, and they are among the 9 unaffected files.

**What we could not determine:** the cause. We can see that affected and unaffected files
split cleanly, but cannot tell from outside when or how the ids diverged.

### 2. The scoring protocol is not recorded

`run_generative_v2.py` scores either by four-way ranking (default) or by pointwise ratings
(`--score_w_ratings`). The saved results record `model`, `model_type` and `chat_template`;
`args.score_w_ratings` is not recorded.

The protocol is recoverable only from the arithmetic of the per-item values — ranking
yields `{0, 1, 0.25}`, ratings yields `(0 in winners)/len(winners)` so `0.5` and `1/3`
appear. By that fingerprint, of the 14 published generative entries with per-item files,
12 were scored by ratings and 2 by ranking, and 0 of 14 record which.

The two are different tasks with different difficulty, so a reader comparing entries in
that column is comparing across it without being able to see that. Adding the flag to
`results_grouped` would make them distinguishable without the fingerprint.

### Reproduction

```
git clone https://github.com/sixijsu77-hub/themis-judge-reliability
cd themis-judge-reliability
pip install huggingface_hub
python scripts/audit_published_results.py # census of all 188 files
python scripts/audit_published_results.py --limit 20 # quick sample instead
```

The full census output is committed at `results/audit/published_results_audit.txt`, so the
counts above can be checked without re-downloading the files.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start with scripts/audit_published_results.py and the committed results/audit/published_results_audit.txt, then inspect eval-set-scores/.json and run_generative_v2.py. Trace how results_grouped is written and compare affected and unaffected files, including the score protocol metadata. Done means the Factuality ids are correct, the scoring mode is recorded, and the audit reproduces complete joins across the published files.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
data, machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.