Add offline score-only / replay path: rescore a predictions JSONL without a model

## Ask

A way to run a task's scorers and metrics over an existing predictions file, with no model and no GPU. Something like:

```
olmo-eval score -t popqa:chat --predictions path/to/predictions.jsonl
```

or equivalently a replay provider (`provider.kind=replay`, `provider.predictions=`) so the whole runner/metric path is exercised without inference.

oe-eval already has this as `--recompute-metrics` ("Recompute metrics for cached predictions"). olmo-eval has no equivalent: `olmo-eval metrics` only queries stored inference telemetry, and `MockProvider` returns fixed canned responses for unit tests rather than replaying a file.

## Why it's worth it

**1. Port certification becomes nearly free, and much stronger.** Certifying #307/#308/#309 against oe-eval consumed a lot of GPU time comparing *aggregate* scores — a weak test, since T=0.6 sampling at n=500 gives SE ≈ 0.018 and only catches gross errors. The strong test is paired: take oe-eval's own stored generations and score them with olmo-eval's scorer, then diff per-instance verdicts. Sampling noise disappears entirely.

Done retroactively for popqa (Qwen3-8B, oe-eval's stored generations, olmo-eval's `PopQAContainsScorer`):

```
instances : 14267
per-instance agreement : 14267/14267 = 100.0000%
oe-eval aggregate : 0.31289
olmo-eval scorer : 0.31289
paired diff : 0.00000
```

That is a far stronger statement than the z=0.06 aggregate cell on #307, and it cost zero GPU-seconds. It only required hand-rolling a script because there's no supported entry point.

**2. Scorer changes currently require re-running the model to see their effect.** #296 (IFEval determinism) and #314 (IFEval robustness) were both pure scorer fixes. Measuring their impact today means regenerating; with a replay path you'd rescore stored predictions in seconds. Same for anyone tuning an answer-extraction cascade.

**3. It's the natural regression test for scorers.** Check in a small predictions fixture per task, assert scores don't move.

## Notes for implementation

- Both harnesses' prediction formats carry what's needed: raw generation, gold/label (oe-eval's popqa `label` is the full alias list), and per-instance metrics for comparison.
- A cross-harness mode (read oe-eval's `task-*-predictions.jsonl` too) would make it a one-command parity gate for the remaining ports — MMLU:cot and LiveCodeBench are still to come.
- Incidental perf note found while doing this by hand: the regex cascade in `common/answer_extraction.py` is slow on long reasoning traces (500 omega items with ~35k-token generations took minutes of pure CPU). Not a correctness issue, but a replay path would make it easy to profile.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx