allenai / allenai/olmo-eval

Corrupted row in VGraf/ifeval_mt breaks length_constraints:nth_paragraph_first_word (row 439)

Open
#325 0 comments 0 reactions 1 assignee Claimed by @VictoriaGraf View on GitHub
Dominant language
Python
Stars
70
Forks
21
Avg merge
2d 11h
Merged PRs (30d)
41

Description

## The row

`VGraf/ifeval_mt`, config `wildchat_unused_withRewrite`, split `test`, index **439**, `key=894000`:

```
prompt: "Rewrite your response. There should be 3 paragraphs. Paragraphs and only
paragraphs are separated with each other by two line breaks. The None-th
paragraph must start with word schedule."
instruction_id_list: ['length_constraints:nth_paragraph_first_word']
kwargs (non-null): {'keyword': 'schedule', 'num_paragraphs': 3}
```

Two independent defects:

1. **Wrong kwarg name.** The verifier accepts `(num_paragraphs, nth_paragraph, first_word)`; the row supplies `keyword`. Almost certainly `first_word` was intended — `'schedule'` is exactly what `first_word` means here.
2. **`nth_paragraph` is missing, and the prompt was rendered with it as `None`** — the instruction literally asks for "The **None-th** paragraph". So the constraint is unsatisfiable as written: no response can comply, and a human reading the prompt can't either.

Scope: 1 instruction of 1,774 rows in `wildchat_unused_withRewrite`. The `ood_wildchat_unused_withRewrite` config is clean (0 of 1,387).

## Effect on evaluation

- Before #314, this row raised `TypeError: unexpected keyword argument 'keyword'` out of `IFEvalScorer.score` and **failed the entire eval run**.
- After #314, it scores as not-followed and records the error in `output.metadata["ifeval"]["errors"]` — safe, and matching oe-eval, which has always try/excepted it to `False`. So historical oe-eval IFBench-MT numbers already include this row as a guaranteed miss for every model.
- Impact on scores is negligible (1/1774 instructions) and identical across models, so it does not distort comparisons — but it is a permanent, invisible 1-instruction penalty in `inst_level_*` metrics.

## Ask

Fix upstream in the dataset rather than in the harness: either supply `first_word` and a real `nth_paragraph` (and re-render the prompt so it doesn't say "None-th"), or drop the row. `VGraf/ifeval_mt` is outside the `allenai` namespace, so this needs whoever owns that dataset.

Filed for tracking; no code change proposed here. #314 makes the harness survive it either way.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.