/review: add a model-family × effort routing table, starting with an A/B baseline vs a minimal prompt
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 28k
- Forks
- 3.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 714
Description
Background
From a side-by-side study of /review against Claude Code's /code-review (binary 2.1.241): their command keeps an internal model family × effort level → prompt cell routing table, with each cell calibrated against an external eval set. The striking data point: on their strongest model family, medium/high effort routes to a minimal prompt (one careful senior-engineer diff pass, ≤15 findings, no angle list, no fan-out, no verify), and on the next-strongest family every level drops the verify subagents. The scaffolding's marginal value demonstrably shrinks as the model gets stronger — to the point of turning negative.
Our 14-agent pipeline was tuned on one model generation, and the routing is a constant: every model gets the same topology.
Proposal
- First, run an A/B baseline — same PR set,
--effort highfull pipeline vs. a minimal prompt ("review this diff as a careful senior engineer would; ≤15 findings; every finding needs a concrete failure scenario"), compare recall and precision per model we ship on. This costs a day and produces the decision datum we currently don't have. - If the gap is significant, keep the pipeline and add a routing table keyed on
config.getModel()family × effort, so weaker/stronger models can get different topologies (e.g. skip verification on models measured to not need it). - If the minimal prompt is competitive on strong models, the 60k-word SKILL.md's maintenance cost needs repricing.
Notes
- The routing table belongs in code (like
parse-args' verdict), not in SKILL.md prose. - DESIGN.md's "LLM call budget" section already tracks per-topology cost; the A/B extends it with per-model quality.
中文摘要
Claude Code 的 /code-review 内部有一张「模型族 × 档位 → 提示词 cell」路由表,每个 cell 用外部评测集标定过。最强模型族的 medium/high 直接路由到极简提示(认真读一遍 diff、≤15 条、无 fan-out、无验证)——脚手架收益随模型能力递减,到最强档已转负。我们的 14-agent 管线是在特定模型上调出来的,路由是常量。建议先做一次 A/B(同批 PR,high 全量 vs 极简提示,比 recall/precision),拿到决策依据;差距显著则加模型路由表,不显著则重新定价 60k 词 SKILL.md 的维护成本。
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read DESIGN.md's “LLM call budget” section, SKILL.md, and the existing /review and parse-args entry points; inspect how config.getModel() is exposed. Start with the proposed A/B comparison of the high-effort pipeline and minimal prompt across shipped models, measuring recall and precision; done means the results support either model-family routing or repricing the current pipeline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100