Generative scores are not reproducible run to run: candidate placement is unseeded

Thanks for RewardBench 2 and for publishing the per-item scores — being able to check
against them is what made the rest of this possible.

### Context

We are building a study of judge robustness, and our own rule is to reproduce a published
number before producing a new one. Three sequence-classifier reward models reproduced
within 0.0094 per subset, agreeing with the published per-item results on 1762/1763,
1760/1763 and 1746/1753 items. Moving to the generative path, we could not establish a
stable baseline at all, and the reason turned out to be in the runner rather than in our
setup.

### What we ran

`allenai/reward-bench` at `05a9005`, unmodified. `Qwen/Qwen2.5-7B-Instruct` via vLLM on one
RTX 4090. The identical command six times at `temperature=0`:

```
python -m scripts.run_generative_v2 --model Qwen/Qwen2.5-7B-Instruct \
--do_not_save --disable_beaker_save --num_gpus 1 --vllm_gpu_util 0.85
```

Three runs reused the on-disk `datasets.map` cache; three ran with a fresh
`HF_DATASETS_CACHE`. Nothing else differed.

### What we got

```
subset run1 run2 run3 cold1 cold2 cold3 spread
Factuality 0.3411 0.3411 0.3411 0.3579 0.3516 0.3747 0.0232
Focus 0.5879 0.5879 0.5879 0.5737 0.5919 0.6162 0.0424
Math 0.5137 0.5137 0.5137 0.3716 0.4590 0.4645 0.0929
Precise IF 0.3375 0.3375 0.3375 0.3125 0.3203 0.3250 0.0125
Safety 0.6800 0.6800 0.6800 0.6711 0.6333 0.6356 0.0378
```

Items whose credited result differs between two runs:

```
reused cache run1 vs run2 : 0 of 1763 = 0.0%
fresh cache cold1 vs cold2: 527 of 1763 = 29.9%
fresh cache cold1 vs cold3: 536 of 1763 = 30.4%
fresh cache cold2 vs cold3: 517 of 1763 = 29.3%
```

The subset score moves by up to 0.0929 between runs of itself, and about 30% of individual
verdicts change while the totals move by a few points, because the flips largely cancel.

`shuffle_option = np.random.randint(0, 4)` appears at three call sites in
`scripts/run_generative_v2.py`, and no seed is set anywhere in the script.

The reused-cache runs matching exactly is not determinism. The draw happens inside a
`datasets.map`, whose results are cached to disk, so on a cache hit the function is not
re-executed and the draw is frozen — across processes, not just within one:

```python
import numpy as np
from datasets import load_dataset # add disable_caching() for the contrast
ds = load_dataset("allenai/reward-bench-2", split="test").select(range(20))
f = lambda x: {**x, "s": int(np.random.randint(0, 4))}
print(ds.map(f)["s"] == ds.map(f)["s"]) # True; False with caching disabled
```

So whether a published generative score reproduces depends on the cache state of the
machine that reran it.

### Relation to #269

#269 argues that the per-category top-two gaps fall inside the sampling noise over items,
using a paired McNemar test on discordant items. That is a different source from this one
and the two add: #269 is about the items being a sample, this is about the same items
producing a different score on a second run of the same model. A comparison between two
generative entries has to clear both.

### Related: unparseable verdicts are credited 0.25

`process_judgement` returns `"error"` when it finds no `[[X]]` marker, and
`process_shuffled` maps that to `0.25` — the expected value of guessing under a four-way
choice. In ranking mode `0.25` can arise no other way, so its frequency is exactly the
parse-failure rate. Two of the 14 published generative entries with per-item files were
scored by ranking, and the higher of those is 352 of 1763 (20.0%) for
`google/gemini-2.5-pro-preview-05-06`, contributing 0.0499 of its 0.6682 non-Ties score.

In ratings mode `0.25` is ambiguous — it is also what `(0 in winners)/len(winners)` returns
for a four-way tie — so for the other 12 entries the observed rate is an upper bound rather
than a measurement. The highest in the whole set is 20.6% (`google/gemini-1.5-flash-8b`,
ratings), and we cannot say how much of that is failure.

We raise it here because it is a second component of a published number that is not judge
skill. For contrast, the 7B open model above produced 0 unparseable verdicts in 1,763 items
in five of six runs and 1 in the sixth, so the format itself does not look like the
obstacle — though that model ran locally on a different prompt path and the comparison is
not clean.

### What we could not determine

Which letter the judge preferred, and therefore the direction of the position effect. The
runner keeps only the credited score and discards the judgement text and the parsed letter,
so we can report the magnitude but not the direction.

### Possible directions

A seed covering the placement draw would make a single run reproducible. Sweeping the four
placements and averaging would additionally make the number stable rather than merely
repeatable, at 4x cost. Reporting the parse-failure rate alongside the score would make the
second component visible. We do not have a view on which fits your plans.

### Reproduction and data

Per-item records for all six runs, and the scripts that regenerate every table above:

https://github.com/sixijsu77-hub/themis-judge-reliability

- `results/variance/` — raw per-item results, one file per run
- `scripts/summarize_variance.py` — regenerates both tables
- `docs/findings/0001-published-results-reproducibility.md` — full write-up

We are running open-weight generative judges for our own study; if the numbers are useful
to you we will share them.