v2 board condition: move every model to tool_choice auto

Why

Under the canonical forced tool_choice, Anthropic's thinking-by-default models skip thinking while every other reasoning-by-default provider reasons regardless — verified and measured in sensitivity/claude-thinking-2026-08.md (Claude Opus 5: 79.8 forced vs 85.6 under auto, 1,984/1,984 parsed both ways).

The harness already accommodates this exact incompatibility when it fails loudly: Moonshot rejects forced tool_choice with thinking enabled, so Kimi models run the JSON contract and reason freely. Anthropic's version fails silently, so it never got the same treatment. Like cases should get like treatment — and tool_choice: auto is less configuration than forcing, so the provider-defaults principle gets stronger, not weaker.

This also aligns the board with field practice: Artificial Analysis and Aider publish reasoning configurations as labeled entries, and MLPerf's closed/open division split is the archetype for a fixed-condition board plus documented variants. The Evals-Consensus reporting-standards effort (evals-consensus.ai) is pushing the same direction on configuration disclosure.

Gates

• Roster-wide probes confirming every model reliably emits the tool call under tool_choice: auto (Opus 5 went 1,984/1,984; the rest are unverified). Any model that answers in text instead is the blocker to resolve or document.
• Ships as a versioned re-run of the full roster with a new dataset version — never an edit to existing scores. The current board stays published as-is.

Roster (decided 2026-08-12)

v2 runs the latest model per line — 20 models. Drops from v1's 31: grok-4.5, gemini-3.6-flash, claude-opus-4.8, claude-opus-4.7, claude-sonnet-4.6, gpt-5.5, gemini-3.5-flash, gemini-3-flash-preview, grok-4.3, kimi-k2.6, qwen-3.7-max.

Keeps: claude-fable-5, claude-opus-5, claude-sonnet-5, claude-haiku-4.5 (latest haiku), gpt-5.6-sol/terra/luna, gpt-5.4-mini and gpt-5.4-nano (latest of their size tiers), gemini-3.7-flash (replaces 3.6-flash as latest flash, added to v1 board 2026-08-17), gemini-3.1-pro-preview, gemini-3.1-flash-lite-preview, grok-4.6 (replaces 4.5 as latest Grok, added to v1 board 2026-08-22), grok-build-0.1, kimi-k3, glm-5.2, qwen3.8-max, deepseek-v4-pro, minimax-m3, inkling.

Note: dropping qwen-3.7-max removes the successor-scored-below-predecessor pair from the board. v1 stays published with both.

Release shape (decided 2026-08-12): combined

v2 ships as one release: fresh households from the stable microcosm build, plus the tool_choice auto condition, on the corrected pipeline. Expected gate opening: the microcosm artifact stabilizes around 2026-08-15. The three Claude thinking runs become the published bridge rows (v1 households + v2 condition). The earlier v2/v3 split below is superseded; kept for the record.

Release split (2026-08-09, superseded)

The two change axes ship as separate versions so each release is decomposable against the last:

• v2 — condition fix, expedited: tool_choice: auto for every model, whole-scenario for the formerly chunked grandfathers, and the harness-audit scoring fixes — on the SAME households and references as v1. No upstream dependency; the three Claude thinking sensitivity runs are already valid v2 rows. Target: as soon as probes and reruns complete.
• v3 — microdata refresh: fresh household draw from the corrected microcosm release, references regenerated on then-current policyengine-us (retiring #92 and the #111 semantics queue), new weighting population — same condition as v2. Gated on the microcosm release.

The interim site note (leaderboard callout + methodology corrections) covers the gap until v2 ships.

Scope expansion (2026-08-08)

A full redo also refreshes the underlying microdata (Max). v2 therefore re-derives everything, not just the request condition:

• Households: a fresh scenario draw from the corrected microcosm sparse release (the successor to the June populace build the current 100 households come from). Gated on that release landing.
• References: regenerated on then-current policyengine-us, which also retires the queued scenario fixes (#92 scenario_121 SC SNAP encoding, the #111 semantics items) that were waiting for the next pinned-version regen.
• Condition: tool_choice: auto for every model, per below.
• Pipeline: the harness-audit fixes land before the rerun so v2 is scored once on the corrected pipeline — filed as #143 (repair overwrite), #144 (trailer override), #145 (explanation-contract policy decision), #146 (resume identity), #147 (cost/latency accounting), #148 (serialization/export/batch/cache bundle).

Because households and condition both change, v1 and v2 rows are not comparable at all — the bridge is the sensitivity family: the three Claude thinking runs already measure v1-households + v2-condition, and keeping one or two non-Claude bridge runs in that cell would let us decompose condition effects from scenario effects.

Probe data point: under auto, Claude Sonnet 5 failed to emit a parseable tool call on 56/1,984 answers (Fable 5 and Opus 5: 0/1,984). "Reliable tool calling under auto" is therefore not universal even within one provider — v2 needs the parse-failure policy stated up front (score-as-miss, matching how JSON-contract parse failures are treated today).

Checklist

• Corrected microcosm sparse release lands (upstream gate)
• Fresh scenario draw + reference regeneration (retires #92 and the #111 semantics queue)
• Run-time budget-escalation on finish=length merged before the rerun — #151 (8b2f195) closes #150
• Refresh the /expand hero stats (best-model exact, owed-SNAP, Medicaid misclassification) from the v2 board at publish
• Harness-audit scoring fixes merged before the rerun — #149 (ea67093) closes #143 #144 #146 #147 #148; #145 stays open as the explanation-policy decision
• Probe all roster models under tool_choice: auto (tool-call reliability + cost/latency deltas)
• Decide contract for any model that fails the probe
• Full-roster re-run under the v2 condition
• Per-row reasoning/serving-configuration labels on the site (not only the paper's Table 2)
• Labeled sensitivity/variant rows surface (e.g. "Claude Opus 5 (thinking)" while v1 remains canonical)
• Paper refreeze with the v2 condition documented and the v1→v2 bridge explained

🤖 Generated with Claude Code

Start with sensitivity/claude-thinking-2026-08.md and the issue checklist, then review the harness-audit work in #143–#149 and the open explanation-policy decision in #145. The work is done when roster-wide auto-condition probes and the contract for failures are settled, the corrected full-roster rerun is published as a new dataset version, and the listed configuration and sensitivity labels are present.