allenai / allenai/reward-bench
eval-set-scores: first ten Factuality ids are wrong in 179 of 188 files; scoring protocol is not recorded
- Ngôn ngữ chính
- Python
- Star
- 736
- Fork
- 99
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Two things about `allenai/reward-bench-2-results` that we hit while joining our own runs
against the published per-item scores.
### 1. Wrong ids in the first ten Factuality entries
`eval-set-scores/.json` stores parallel arrays, so `(subset, id)` is the only way to
line an item up against anything else. In **179 of 188 files** the first ten Factuality
entries are labelled `30`–`39` instead of `0`–`9`, producing ten duplicate keys per file.
```
first ten Factuality ids, affected : ['30','31','32','33','34','35','36','37','38','39']
expected : ['0','1','2','3','4','5','6','7','8','9']
```
All 179 affected files show the identical shape. The 9 unaffected ones are the seven
Skywork-Reward-V2 models, `Skywork/Skywork-VL-Reward-7B`, and `HFXM/RAMO-Llama3.1-8B`.
Subset totals are unaffected: non-Ties scoring takes a mean and never reads the id, and the
Ties subset, which does read it, is not among the affected entries. What breaks is joining
on id — ten items attach to the wrong row and nothing raises.
We ran into this before we went looking for it. Validating our harness against
`LxzGordon/URM-LLaMa-3.1-8B`, our per-item join covered 1753 of 1763 items rather than all
of them, and within Factuality 465 of 475 — ten short in both cases, matching the ten
duplicate keys. The two Skywork-Reward-V2 models we validated alongside it joined
completely, and they are among the 9 unaffected files.
**What we could not determine:** the cause. We can see that affected and unaffected files
split cleanly, but cannot tell from outside when or how the ids diverged.
### 2. The scoring protocol is not recorded
`run_generative_v2.py` scores either by four-way ranking (default) or by pointwise ratings
(`--score_w_ratings`). The saved results record `model`, `model_type` and `chat_template`;
`args.score_w_ratings` is not recorded.
The protocol is recoverable only from the arithmetic of the per-item values — ranking
yields `{0, 1, 0.25}`, ratings yields `(0 in winners)/len(winners)` so `0.5` and `1/3`
appear. By that fingerprint, of the 14 published generative entries with per-item files,
12 were scored by ratings and 2 by ranking, and 0 of 14 record which.
The two are different tasks with different difficulty, so a reader comparing entries in
that column is comparing across it without being able to see that. Adding the flag to
`results_grouped` would make them distinguishable without the fingerprint.
### Reproduction
```
git clone https://github.com/sixijsu77-hub/themis-judge-reliability
cd themis-judge-reliability
pip install huggingface_hub
python scripts/audit_published_results.py # census of all 188 files
python scripts/audit_published_results.py --limit 20 # quick sample instead
```
The full census output is committed at `results/audit/published_results_audit.txt`, so the
counts above can be checked without re-downloading the files.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.