allenai / allenai/reward-bench

eval-set-scores: first ten Factuality ids are wrong in 179 of 188 files; scoring protocol is not recorded

Đang mở
#273 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
736
Fork
99
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

Two things about `allenai/reward-bench-2-results` that we hit while joining our own runs
against the published per-item scores.

### 1. Wrong ids in the first ten Factuality entries

`eval-set-scores/.json` stores parallel arrays, so `(subset, id)` is the only way to
line an item up against anything else. In **179 of 188 files** the first ten Factuality
entries are labelled `30`–`39` instead of `0`–`9`, producing ten duplicate keys per file.

```
first ten Factuality ids, affected : ['30','31','32','33','34','35','36','37','38','39']
expected : ['0','1','2','3','4','5','6','7','8','9']
```

All 179 affected files show the identical shape. The 9 unaffected ones are the seven
Skywork-Reward-V2 models, `Skywork/Skywork-VL-Reward-7B`, and `HFXM/RAMO-Llama3.1-8B`.

Subset totals are unaffected: non-Ties scoring takes a mean and never reads the id, and the
Ties subset, which does read it, is not among the affected entries. What breaks is joining
on id — ten items attach to the wrong row and nothing raises.

We ran into this before we went looking for it. Validating our harness against
`LxzGordon/URM-LLaMa-3.1-8B`, our per-item join covered 1753 of 1763 items rather than all
of them, and within Factuality 465 of 475 — ten short in both cases, matching the ten
duplicate keys. The two Skywork-Reward-V2 models we validated alongside it joined
completely, and they are among the 9 unaffected files.

**What we could not determine:** the cause. We can see that affected and unaffected files
split cleanly, but cannot tell from outside when or how the ids diverged.

### 2. The scoring protocol is not recorded

`run_generative_v2.py` scores either by four-way ranking (default) or by pointwise ratings
(`--score_w_ratings`). The saved results record `model`, `model_type` and `chat_template`;
`args.score_w_ratings` is not recorded.

The protocol is recoverable only from the arithmetic of the per-item values — ranking
yields `{0, 1, 0.25}`, ratings yields `(0 in winners)/len(winners)` so `0.5` and `1/3`
appear. By that fingerprint, of the 14 published generative entries with per-item files,
12 were scored by ratings and 2 by ranking, and 0 of 14 record which.

The two are different tasks with different difficulty, so a reader comparing entries in
that column is comparing across it without being able to see that. Adding the flag to
`results_grouped` would make them distinguishable without the fingerprint.

### Reproduction

```
git clone https://github.com/sixijsu77-hub/themis-judge-reliability
cd themis-judge-reliability
pip install huggingface_hub
python scripts/audit_published_results.py # census of all 188 files
python scripts/audit_published_results.py --limit 20 # quick sample instead
```

The full census output is committed at `results/audit/published_results_audit.txt`, so the
counts above can be checked without re-downloading the files.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.