Implementing lessons from OLMES

The OLMES paper is a pretty interesting read and is complementary to LM-Eval. I think there are a few features we can consider implementing based on lessons and recommendations from the paper.

1. **Implement mode-switch between Multiple-choice formulation and Completion/cloze formulation.** Empirically models respond to both version differently depending on how much tokens trained with the former showing a stronger signal in later stages (400B tokens and above). The recommendation is to evaluate on both and take the highest score. Lm-eval would benefit from having the ability to write 1 prompt format and have it automatically be used in both formulations.
2. **Normalization should be a configuration feature.** We use both non-normalized accuracy and normalized accuracy, specifically by dividing the log-probability by the length or characters. It would be great to be able to choose the normalization and add more (normalization like based on token length or pointwise-mutual information)
3. **Further add support for fewshot selection.** We have some support for hardcoding fewshot samples, but we've never really supported a way to make it easier to select fewshots with conditions like "make sure the answer choices are not all A" or set a list of index number that points to the samples directly.

cc: @haileyschoelkopf @StellaAthena

Start by reading the OLMES paper and comparing its recommendations with the harness's existing evaluation and hardcoded few-shot support. The issue covers three separate capabilities—dual prompt formulations, configurable normalization, and conditional few-shot selection—so “done” would require a scoped decision and validation for whichever part is selected.

Written by the indexing model from the issue text.