feat: pin (or publish) the OpenRouter provider per run; enable prompt caching on the invariant prompt part
- Dominant language
- Python
- Stars
- 106
- Forks
- 11
- PR merge metrics
- No merged PRs in 30d
Description
## 1. Provider variance is folded into model variance
The response logs in the CC0 dataset show `extra_body` with
`provider: {sort: "throughput"}`, which scattered generations across upstreams —
we saw Amazon Bedrock, Google, SiliconFlow, AtlasCloud, and DeepInfra serving
runs of the same model. Upstreams differ in quantization and sampling defaults,
so part of the leaderboard's run-to-run variance is *provider* variance, not
model variance.
Suggestions, in increasing order of effort:
- Surface the served provider (already present in `responses.jsonl`) in the
leaderboard/run metadata.
- Pin `provider.order` / `provider.only` per model for benchmark runs.
## 2. Prompt caching is off — roughly a 3× cost saving available
The ~42k-char invariant part 0 (game manual + strategy) is re-sent on every one
of ~200 calls per game (`cache_control` is unset on all three content parts) —
about 10.5k tokens of identical input per call, which matches the measured ~14k
input tokens/call. The three-part message split is already exactly the right
shape for caching: adding `cache_control` to part 0 (Anthropic-style caching,
which OpenRouter passes through) cuts per-game cost roughly 3× for
caching-capable models (~$5.60/game → under $2 at Sonnet-class pricing), with
no behavioral change.
## 3. Minor: seeded runs are not perfectly reproducible across game launches
Some SMODS-rebuilt pools are iterated in per-launch order before being fed to
the seeded RNG — we confirmed this for To Do List's target hand and Orbital
Tag's hand choice (same seed + identical action trace → different outcome after
a game relaunch, stable within one launch). Worth a caveat wherever the
benchmark claims seed-level determinism.
Contributor guide
Assessment
This issue has not been assessed yet.