Upstream defects found during eval-port certification (IFEval datasets, oe-eval answer extraction)
- Ngôn ngữ chính
- Python
- Star
- 70
- Fork
- 21
- Merge trung bình
- 2 ngày 11 giờ
- Pull request đã merge (30 ngày)
- 41
Mô tả
Four defects found while certifying the IFEval and OMEGA ports against `oe-eval-internal`. None is an olmo-eval bug — all are in datasets or in oe-eval — but each affects numbers produced by both harnesses, so they're worth tracking and reporting upstream.
## 1. `google/IFEval` — two rows specify a `letter` no verifier can honor
| key | `letter` | `let_frequency` | `let_relation` |
|---|---|---|---|
| 1122 | `'#'` | 4 | at least |
| 1129 | `'!'` | 6 | at least |
`LetterFrequencyChecker.build_description` rejects any `letter` outside `a`–`z`:
```python
if not letter or len(letter) > 1 or ord(letter.lower()) < 97 or ord(letter.lower()) > 122:
self._letter = random.choice(list(string.ascii_letters))
```
so it scores a **randomly chosen letter's** frequency instead of the one the prompt asks for. Both `ifbench` and oe-eval's vendored `oe_eval/dependencies/ifeval` do this — it's inherited from the original Google implementation.
Consequence: these two rows are nondeterministic in every IFEval harness. Repeated builds of key 1129 drew `x, c, t, i, c, t` on consecutive calls, flipping the verdict between runs. Bounded at 2/541 prompts (~0.37 points of prompt-level noise), but it means IFEval prompt-level scores are not exactly reproducible and small differences between runs should not be over-read.
The prompts themselves are coherent ("Contain 6 or more exclamation marks"), so the fix is in the verifier — either support non-alphabetic characters or raise instead of silently randomizing.
## 2. `VGraf/ifeval_mt` — one row uses the wrong kwarg name
Tracked in full in #325, which has the complete diagnosis: besides supplying `keyword` where the verifier expects `first_word`, the row also leaves `nth_paragraph` unset and renders the prompt as "The **None-th** paragraph must start with word schedule" — unsatisfiable as written. 1 of 1,774 instructions; the ood config is clean at 0 of 1,387.
## 3. oe-eval `extract_answer` — empty match wins the raw fallback
`oe_eval/tasks/utils.py`:
```python
ans_match = re.findall(answer_regex, continuation)
if ans_match:
answer_string = ans_match[-1] if use_last_raw_match else ans_match[0]
```
With OMEGA's fallback regex `(.*)\.?`, an unformatted continuation `42` yields `['42', '']` — unanchored, so it also matches at end-of-string — and `[-1]` selects the empty string. Both `exact_match` and `exact_match_flex` then reject exactly the bare-answer responses the fallback exists to accept.
Verified identical on both harnesses: input `42` → `('', 0.1)`.
## 4. oe-eval `omega_500` — colon-bearing answer prefixes are unreachable
`answer_prefix_regexes` orders `"(?i)the answer is"` before `"(?i)answer is:"`, and extraction returns after the first matching prefix. For `The answer is: 42` the shorter pattern matches first, leaving `: 42`, and the catch-all answer regex extracts the leading colon.
Verified identical on both harnesses: `The answer is: 42` → `(': 42', 0.5)`.
---
3 and 4 were surfaced by Codex review on #308 and are correct findings. They were **not** changed in that PR: olmo-eval's `common/answer_extraction.py` is a deliberate verbatim port, and diverging would break the per-instance parity the migration is being certified on (currently 500/500 on both models). Fixing them upstream lets both harnesses move together.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Đánh giá
Issue này chưa được đánh giá.