aio-libs / aio-libs/aiobotocore

Revisit Sonnet-vs-Opus split in the Claude workflows now that the price gap has narrowed

未關閉
#1,660 2 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
1.4k
分支
209
平均合併
4 天 15 小時
30 天內合併 PR
16

描述

## Background

The model choices in our Claude-driven workflows were made when Opus was \$15/\$75 per million input/output tokens — roughly a 5–10x premium over Sonnet. The comments justifying those choices still assume that gap:

- `.github/workflows/claude.yml` — "the botocore-sync classifier that already runs on Sonnet at ~\$0.10 per run vs ~\$1+ for Opus"
- `.github/workflows/botocore-sync.yml` — the same "~\$0.10 versus ~\$1+" claim for the classifier job
- `.github/workflows/botocore-sync.yml` — "no-port → Sonnet (mechanical bump+PR; ~5× cheaper output)"

That premium no longer exists. Opus 4.8 is \$5/\$25 against Sonnet 5's \$3/\$15 — about 1.67x on output, and 2.5x while Sonnet 5's introductory \$2/\$10 rate runs (through 2026-08-31).

## The open question

`plugins/aiobotocore-bot/evals/_common.py` already absorbed the new pricing and picked Opus for the evals, noting the drop narrowed the effective gap to Sonnet+thinking to ~25% and that the accuracy edge (8/8 vs 7/8 on the regression suite) makes Opus the better cost/quality point.

But that file's comment also claimed Opus was the "default across eval, sync, and reviewer workflows," which isn't true:

| Workflow | Model today |
|-|-|
| `claude.yml` (PR review, `@claude`) | `sonnet` |
| `botocore-sync.yml` classify | `sonnet` |
| `botocore-sync.yml` sync | `opus`, or `sonnet` on a `no-port` verdict |
| `draft-release.yml` | `sonnet` |
| evals (`_common.py`) | `claude-opus-4-8` |

So the repo is internally inconsistent about which model policy it is on. Worth deciding deliberately rather than by drift.

## Considerations

**For moving the reviewer to Opus:** the original cost case was built on a ~10x gap that is now ~1.67x. The evals already show a measurable accuracy edge for Opus on the closest analogous task.

**Against:** PR-review traffic is the highest-volume Claude job in the repo — every push to every open PR triggers a run — so even a 1.67x multiplier compounds across far more runs than the sync or release jobs. Code review is also the most mechanical of the tasks (diff-walk plus a project-rules check against `CLAUDE.md`), which is what put it in Sonnet's range originally.

**Possible middle ground:** route by prompt content or PR label, so `@claude` tasks that need deeper reasoning escalate to Opus while routine push-triggered reviews stay on Sonnet. `claude.yml` already floats this idea in a comment.

## Suggested next step

Run the existing eval suite against the review path on both models to get a real accuracy delta, then price it against actual monthly PR-review volume before deciding.

## Related

The stale-comment cleanup and the eval model bump (Opus 4.7 → 4.8, Sonnet 4.6 → 5) are being handled separately; this issue covers only the routing decision.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。