QwenLM / QwenLM/qwen-code

/review: add a model-family × effort routing table, starting with an A/B baseline vs a minimal prompt

Open
#9,783 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

category/tools priority/P0 type/enhancement
Dominant language
TypeScript
Stars
28k
Forks
3.1k
Avg merge
1d 2h
Merged PRs (30d)
714

Description

Background

From a side-by-side study of /review against Claude Code's /code-review (binary 2.1.241): their command keeps an internal model family × effort level → prompt cell routing table, with each cell calibrated against an external eval set. The striking data point: on their strongest model family, medium/high effort routes to a minimal prompt (one careful senior-engineer diff pass, ≤15 findings, no angle list, no fan-out, no verify), and on the next-strongest family every level drops the verify subagents. The scaffolding's marginal value demonstrably shrinks as the model gets stronger — to the point of turning negative.

Our 14-agent pipeline was tuned on one model generation, and the routing is a constant: every model gets the same topology.

Proposal

  1. First, run an A/B baseline — same PR set, --effort high full pipeline vs. a minimal prompt ("review this diff as a careful senior engineer would; ≤15 findings; every finding needs a concrete failure scenario"), compare recall and precision per model we ship on. This costs a day and produces the decision datum we currently don't have.
  2. If the gap is significant, keep the pipeline and add a routing table keyed on config.getModel() family × effort, so weaker/stronger models can get different topologies (e.g. skip verification on models measured to not need it).
  3. If the minimal prompt is competitive on strong models, the 60k-word SKILL.md's maintenance cost needs repricing.

Notes

  • The routing table belongs in code (like parse-args' verdict), not in SKILL.md prose.
  • DESIGN.md's "LLM call budget" section already tracks per-topology cost; the A/B extends it with per-model quality.
中文摘要

Claude Code 的 /code-review 内部有一张「模型族 × 档位 → 提示词 cell」路由表,每个 cell 用外部评测集标定过。最强模型族的 medium/high 直接路由到极简提示(认真读一遍 diff、≤15 条、无 fan-out、无验证)——脚手架收益随模型能力递减,到最强档已转负。我们的 14-agent 管线是在特定模型上调出来的,路由是常量。建议先做一次 A/B(同批 PR,high 全量 vs 极简提示,比 recall/precision),拿到决策依据;差距显著则加模型路由表,不显著则重新定价 60k 词 SKILL.md 的维护成本。

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read DESIGN.md's “LLM call budget” section, SKILL.md, and the existing /review and parse-args entry points; inspect how config.getModel() is exposed. Start with the proposed A/B comparison of the high-effort pipeline and minimal prompt across shipped models, measuring recall and precision; done means the results support either model-family routing or repricing the current pipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.