Feature Request: Adaptive Item Selection for Expensive Generative Evaluations

## Summary

I propose adding **adaptive item selection** to OLMES for expensive generative evaluations (e.g., judge-based tasks like AlpacaEval). Instead of evaluating all items, an adaptive evaluator would dynamically select which items to score based on the model's current performance, reducing compute cost while maintaining reliability.

**This is intended as an opt-in development tool; default suites remain unchanged.** Official reporting would continue using static, full-coverage evaluation for reproducibility and cross-model comparability.

This proposal is a follow-up to my OLMo-core issue ([#491](https://github.com/allenai/OLMo-core/issues/491)), where @tyler-romero and @davidheineman suggested that adaptive evaluation belongs in OLMES generative evals rather than in-loop training evaluation.

**Note**: This is not adaptive task selection. The task suite remains fixed; adaptivity is only within a chosen task's items, and only in opt-in dev mode.

## Motivation

Recent research demonstrates that adaptive evaluation can significantly improve efficiency:

- **Fluid LM Benchmarking** (Hofmann et al., 2025) shows that adapting question selection to each model can achieve **~50× fewer items on MMLU with lower variance** using item-response theory for dynamic selection ([arXiv:2509.11106](https://arxiv.org/abs/2509.11106)).
- **Signal and Noise** (Heineman et al., 2025) shows that filtering out too-easy or too-unstable items improves benchmark reliability ([arXiv:2508.13144](https://arxiv.org/abs/2508.13144)).

For OLMES, the strongest value proposition is **expensive generative tasks with external API calls** (e.g., AlpacaEval-style judge evaluations, safety/jailbreak suites). OLMES already uses OpenAI judges for AlpacaEval and SimpleQA in their suite instructions (requiring `OPENAI_API_KEY`), making these tasks costly per item. Adaptive subsampling is attractive for development workflows where faster feedback is more valuable than exhaustive coverage.

## Proposed Solution

### Core Concept: Anchor Set + Adaptive Selection

A simple pattern that addresses comparability concerns:

1. **Evaluate a fixed anchor subset** (e.g., 100 items) for cross-model comparability
2. **Spend remaining budget adaptively** on items that are most informative for the current model

This keeps a stable reference point while still capturing Fluid-style efficiency gains. The run logs anchor IDs + adaptively selected IDs + seed + policy config so the exact subset can be replayed.

### MVP Target

**Focus on one expensive generative task for MVP**: AlpacaEval-style judge evaluations (e.g., `alpaca_eval_v2::tulu` or similar). These tasks:
- Use external API calls (expensive; requires `OPENAI_API_KEY` for judge evaluations)
- Have clear per-item outcomes (win/loss from judge)
- Benefit from adaptive selection (items vary in informativeness across models)

### Implementation Approach (High-Level)

The adaptive evaluator would:

- **Operate at the item level within a task** (aligned with Fluid's approach)
- **Leverage OLMES's existing instance-level logging** (logprobs, detailed predictions) to estimate item difficulty and informativeness
- **Use a pluggable selection policy** (e.g., informativeness-based for MVP, IRT-based later)
- **Integrate with existing OLMES pipeline** (`Task` → instances → `evaluate` → metrics) without requiring changes to core components

**Design note**: The MVP would be batch-adaptive (adapt between batches, parallel within a batch), so OLMES can still shard instances across workers/hosts for each batch; only the selection step is sequential.

Since OLMES already records detailed instance-level predictions (logprobs, etc.), informativeness proxies for adaptive selection (e.g., uncertainty, self-consistency, response entropy, disagreement across sampling) are readily available.

### Example Usage (Conceptual / Proposed)

```bash
olmes \
--model allenai/OLMo-2-0425-7B \
--task '{"task_name":"alpaca_eval_v2::tulu","adaptive_eval":true,"adaptive_anchor_size":100,"adaptive_max_items":500}' \
--output-dir workspace \
--dry-run
```

(Exact config structure here is **conceptual**; I would follow existing OLMES task naming and config patterns.)

This would evaluate 100 fixed anchor items (for comparability) plus up to 400 adaptively selected items.

## Academic Background

1. **Fluid LM Benchmarking: Adapting Language Model Evaluation to Each Model** (Hofmann et al., 2025)
- [arXiv:2509.11106](https://arxiv.org/abs/2509.11106)
- Demonstrates ~50× fewer items on MMLU with lower variance using item-response theory for dynamic question selection
- Operates at item level within tasks (not task-level selection)

2. **Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation** (Heineman et al., 2025)
- [arXiv:2508.13144](https://arxiv.org/abs/2508.13144)
- Shows that filtering too-easy or too-unstable items improves reliability
- Demonstrates better signal-to-noise ratio leads to more reliable decision-making

## Implementation Plan

### Phase 1: MVP (Informativeness-Based Selection)

- Implement `AdaptiveGenerativeEvaluator` that wraps existing OLMES evaluation pipeline
- Add simple `SelectionPolicy` interface with `InformativenessBandPolicy` implementation
- Use OLMES's existing instance-level logprobs/predictions to estimate item difficulty and **informativeness proxies** (e.g., uncertainty, self-consistency, response entropy, disagreement across sampling) even when the external judge does not expose confidence
- Support anchor set + adaptive remainder pattern
- Opt-in via task config or CLI flag
- Target: AlpacaEval-style judge task

### Phase 2: IRT-Based Selection (Optional)

- Add `IRTPolicy` using pre-computed item difficulty parameters
- Information-gain-based item selection (Fluid-style)
- Can leverage Fluid's released code/assets if available