coder / coder/balatrollm

feat: pin (or publish) the OpenRouter provider per run; enable prompt caching on the invariant prompt part

Open
#86 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
106
Forks
11
PR merge metrics
No merged PRs in 30d

Description

## 1. Provider variance is folded into model variance

The response logs in the CC0 dataset show `extra_body` with
`provider: {sort: "throughput"}`, which scattered generations across upstreams —
we saw Amazon Bedrock, Google, SiliconFlow, AtlasCloud, and DeepInfra serving
runs of the same model. Upstreams differ in quantization and sampling defaults,
so part of the leaderboard's run-to-run variance is *provider* variance, not
model variance.

Suggestions, in increasing order of effort:

- Surface the served provider (already present in `responses.jsonl`) in the
leaderboard/run metadata.
- Pin `provider.order` / `provider.only` per model for benchmark runs.

## 2. Prompt caching is off — roughly a 3× cost saving available

The ~42k-char invariant part 0 (game manual + strategy) is re-sent on every one
of ~200 calls per game (`cache_control` is unset on all three content parts) —
about 10.5k tokens of identical input per call, which matches the measured ~14k
input tokens/call. The three-part message split is already exactly the right
shape for caching: adding `cache_control` to part 0 (Anthropic-style caching,
which OpenRouter passes through) cuts per-game cost roughly 3× for
caching-capable models (~$5.60/game → under $2 at Sonnet-class pricing), with
no behavioral change.

## 3. Minor: seeded runs are not perfectly reproducible across game launches

Some SMODS-rebuilt pools are iterated in per-launch order before being fed to
the seeded RNG — we confirmed this for To Do List's target hand and Orbital
Tag's hand choice (same seed + identical action trace → different outcome after
a game relaunch, stable within one launch). Worth a caveat wherever the
benchmark claims seed-level determinism.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.