github / github/copilot-cli

rubber-duck: model-emitted `model` argument silently overrides the complementary strategy and the user's /subagents setting

Đang mở
#4,432 2 bình luận 0 reaction 0 người được giao Xem trên GitHub

Chưa có ai nhận issue này.

triage
Ngôn ngữ chính
Shell
Star
11.2k
Fork
1.9k
Merge trung bình
14 giờ 16 phút
Pull request đã merge (30 ngày)
6

Mô tả

Summary

The rubber-duck sub-agent exists to give a cross-family second opinion — a Claude session gets a GPT reviewer, and vice versa. Its shipped definition deliberately omits model: so the complementary strategy can select an opposite-family model.

However, the task tool exposes an optional model parameter to the calling model. When the parent agent supplies it, that value wins over everything, including a complementary value the user explicitly configured in /subagents. The result is a Claude session getting a Claude reviewer — silently, with no warning, defeating the agent's entire purpose.

Version: GitHub Copilot CLI 1.0.79 (darwin-arm64)

Why this is a defect rather than expected precedence

"Explicit argument beats config file" is correct precedence when a human supplied the argument. Here the argument is a non-deterministic token sequence emitted by an LLM that has no knowledge that the user configured anything. So model discretion silently outranks explicit user configuration, and nothing surfaces the divergence.

Contributing factors:

  1. The design intent is documented in the shipped agent file. definitions/rubber-duck.agent.yaml:

    # model: omitted - will be selected dynamically at runtime based on user's current model preference
    
  2. No guidance is given to the calling model. The task tool describes model generically ("Use 'model' parameter to override the default model (20 models available)"). Neither that description nor the rubber-duck agent description mentions that setting model defeats the complementary strategy. The calling model has no signal that this agent is special.

  3. No guard and no warning exist. Searching the native runtime.node for complementary-related user-facing strings turns up only the unavailable error ("...uses a complementary model... but none is available..."). There is no warning when an explicit model collapses the agent to the parent's own family.

Reproduction

Observed in a real session (claude-opus-5 parent). From ~/.copilot/session-state/<id>/events.jsonl:

tool.execution_start, toolCallId: toolu_01DXhyxFyFHbcsQmGQseBwmi:

{
  "agent_type": "rubber-duck",
  "name": "reval-review",
  "description": "Review README revalidation changes",
  "model": "claude-opus-4.8",
  "prompt": "..."
}

Corresponding subagent.completed:

agentName: "rubber-duck"   model: "claude-opus-4.8"

A Claude session received a Claude reviewer.

Verified precedence

Calling the native resolver directly (runtime.nodesubagentStartPlan), parent model claude-opus-5, available models claude-opus-5 / claude-opus-4.8 / gpt-5.6-sol all at price category high:

/subagents setting explicit model arg resolved model
(none) (none) gpt-5.6-sol
(none) claude-opus-4.8 claude-opus-4.8
complementary claude-opus-4.8 claude-opus-4.8
gpt-5.6-sol claude-opus-4.8 claude-opus-4.8
gpt-5.6-sol (none) gpt-5.6-sol

Effective precedence:

explicit `model` arg  >  /subagents setting  >  agent-declared model  >  complementary  >  inherit

Rows 3 and 4 are the problem: a setting the user deliberately chose is overridden by a value the model guessed.

Repro script
// from ~/.copilot/pkg/darwin-arm64/1.0.79/
const h = require("./prebuilds/darwin-arm64/runtime.node");
const models = [
  { id: "claude-opus-5",   label: "C5",  priceCategory: "high" },
  { id: "claude-opus-4.8", label: "C48", priceCategory: "high" },
  { id: "gpt-5.6-sol",     label: "Sol", priceCategory: "high" },
];
const plan = (o) => JSON.parse(h.subagentStartPlan(JSON.stringify({
  agentType: "rubber-duck",
  availableBuiltinNames: ["rubber-duck"],
  customAgentNames: [], isCustom: false,
  explicitModel: o.explicitModel,
  subagents: o.subagents,
  selectedParentModel: "claude-opus-5",
  autoMode: false,
  defaultToComplementary: true,
  complementarySelection: { model: "gpt-5.6-sol" },
  availableModels: models,
})));

const S = (m) => ({ agents: { "rubber-duck": { model: m } } });
console.log(plan({}));                                                        // gpt-5.6-sol
console.log(plan({ explicitModel: "claude-opus-4.8" }));                      // claude-opus-4.8
console.log(plan({ explicitModel: "claude-opus-4.8", subagents: S("complementary") })); // claude-opus-4.8
console.log(plan({ explicitModel: "claude-opus-4.8", subagents: S("gpt-5.6-sol") }));   // claude-opus-4.8

Expected behaviour

Any one of these would resolve it; they are listed cheapest first.

  1. Document the constraint to the calling model. Add to the task tool description, or the rubber-duck agent description: "Do not set model for this agent — it selects a complementary, opposite-family model automatically." Low-risk and likely fixes the majority of occurrences.

  2. Rank a configured /subagents value above a model-emitted model argument. Distinguish a human per-invocation override from one the model produced, and let explicit user configuration win. This is the principled fix.

  3. Warn on family collapse. When rubber-duck resolves to the same model family as the parent session, surface it in the timeline or emit a warning. Today the only way to discover this is to read events.jsonl by hand.

Impact

The failure is silent and the output is still plausible — a same-family reviewer returns a confident critique that simply shares the author model's blind spots. Users believe they are getting a cross-family second opinion when they are not, and the /subagents setting they configured to guarantee it does not hold.

Current workaround

Instruct the parent agent never to pass model for agent_type: "rubber-duck" (e.g. in ~/.copilot/copilot-instructions.md). This is per-machine and depends on instruction adherence, so it is not a substitute for a product fix.

Related

#4380 reports the same user-visible symptom (rubber-duck running the parent's own model family), but appears to be a different mechanism: there the reporter's /subagents shows complementary — unavailable, disabled with Overridden: No, i.e. no opposite-family model was available at all.

In the case above, complementary was available (gpt-5.6-sol was present at the same price category and is what resolves when no model argument is passed) and was overridden by the explicit argument. So the two may need separate fixes, and I have added this evidence there as well.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Hướng nghiên cứu

Bắt đầu với subagentStartPlan trong runtime.node gốc và definitions/rubber-duck.agent.yaml được cung cấp; so sánh explicitModel, subagents và complementarySelection bằng script tái hiện được cung cấp. Đồng thời kiểm tra công cụ task và các mô tả agent. Được coi là hoàn tất khi bản sửa được chọn ngăn chặn hoặc làm lộ việc phân giải im lặng trong cùng một họ, đồng thời vẫn giữ nguyên hành vi bổ trợ đã được ghi trong tài liệu.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
javascript, node.js
Lĩnh vực
cli, developer-experience
Loại issue
Lỗi
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
48/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.