Reproduction of Tulu-3-8B-SFT on MMLU

Hi, thanks for great tool for reproducing evaluation of Ai2 models!

I ran the olmes eval with:
```
olmes \
--model allenai/Llama-3.1-Tulu-3-8B-SFT \
--model-type vllm \
--model-args '{"gpu_memory_utilization": 0.8, "max_length": 4096}' \
--task tulu_3_dev \
--output-dir out \
--batch-size auto
```
and I was able to reproduce most of the benchmarks with <1% difference from the results reported in the [Tulu3 paper](https://arxiv.org/pdf/2411.15124). However, on MMLU I have a discrepancy of 4% which is concerning. See the table below where for reproduction I report `primary_score` for all of the tasks (expect of MATH where I consider `exact_match_flex_macro`).

| Benchmark | Tülu 3 SFT | Reproduction |
|----------------------------|------------|--------------|
| MMLU (0 shot, CoT) | **65.9** | **62.0** |
| PopQA (15 shot) | 29.3 | 29.5 |
| TruthfulQA (6 shot) | 46.8 | 46.8 |
| BigBenchHard (3 shot, CoT) | 69.7 | 69.6 |
| DROP (3 shot) | 61.3 | 61.2 |
| MATH (4 shot CoT, Flex) | 31.5 | 31.0 |
| GSM8K (8 shot, CoT) | 76.2 | 76.1 |
| HumanEval (pass@10) | 86.2 | 87.3 |
| HumanEval+ (pass@10) | 81.4 | 80.7 |
| IFEval (prompt loose) | 72.8 | 72.6 |

Do you know what may be a source of such difference in results? Thanks in advance