Corrupted row in VGraf/ifeval_mt breaks length_constraints:nth_paragraph_first_word (row 439)
- Dominant language
- Python
- Stars
- 70
- Forks
- 21
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 41
Description
## The row
`VGraf/ifeval_mt`, config `wildchat_unused_withRewrite`, split `test`, index **439**, `key=894000`:
```
prompt: "Rewrite your response. There should be 3 paragraphs. Paragraphs and only
paragraphs are separated with each other by two line breaks. The None-th
paragraph must start with word schedule."
instruction_id_list: ['length_constraints:nth_paragraph_first_word']
kwargs (non-null): {'keyword': 'schedule', 'num_paragraphs': 3}
```
Two independent defects:
1. **Wrong kwarg name.** The verifier accepts `(num_paragraphs, nth_paragraph, first_word)`; the row supplies `keyword`. Almost certainly `first_word` was intended — `'schedule'` is exactly what `first_word` means here.
2. **`nth_paragraph` is missing, and the prompt was rendered with it as `None`** — the instruction literally asks for "The **None-th** paragraph". So the constraint is unsatisfiable as written: no response can comply, and a human reading the prompt can't either.
Scope: 1 instruction of 1,774 rows in `wildchat_unused_withRewrite`. The `ood_wildchat_unused_withRewrite` config is clean (0 of 1,387).
## Effect on evaluation
- Before #314, this row raised `TypeError: unexpected keyword argument 'keyword'` out of `IFEvalScorer.score` and **failed the entire eval run**.
- After #314, it scores as not-followed and records the error in `output.metadata["ifeval"]["errors"]` — safe, and matching oe-eval, which has always try/excepted it to `False`. So historical oe-eval IFBench-MT numbers already include this row as a guaranteed miss for every model.
- Impact on scores is negligible (1/1774 instructions) and identical across models, so it does not distort comparisons — but it is a permanent, invisible 1-instruction penalty in `inst_level_*` metrics.
## Ask
Fix upstream in the dataset rather than in the harness: either supply `first_word` and a real `nth_paragraph` (and re-render the prompt so it doesn't say "None-th"), or drop the row. `VGraf/ifeval_mt` is outside the `allenai` namespace, so this needs whoever owns that dataset.
Filed for tracking; no code change proposed here. #314 makes the harness survive it either way.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.