CommandCodeAI / CommandCodeAI/command-code

Tool calls silently fail on strict OpenAI-wire BYOK providers: web_search/MCP tools announced in text or empty completion instead of tool_use — undeclared-tool progressive disclosure not tolerated by all servers

未關閉
#787 1 則留言 0 個 reaction 已指派 1 人 在 GitHub 檢視

@ahmadbilaldev 已經在處理了。

開始於 2026年9月17日。

主要語言
沒有語言資料
星號
4k
分支
350
PR 合併指標
30 天內沒有已合併 PR

描述

Summary

Command Code uses progressive tool disclosure: most tool schemas (web_search, web_fetch, all mcp__* tools) are not declared in the request's tools array. Their schemas are delivered lazily as the text result of a search_tools call, and the model is expected to emit a tool_calls block for a tool that is not present in the OpenAI tools parameter of that request.

Some OpenAI-wire serving stacks tolerate this (emitting tool_calls for undeclared, in-prompt-described tools); others — in our case NeuralWatt's GLM serving (glm-5.3-flash, glm-5.2) and Openrouter — strictly constrain emission to declared tools. On those servers the model either writes a text reply announcing the call ("Now I'll run the web search…") with finish_reason: "stop" and zero tool_calls deltas, or returns a completely empty completion. The harness's continuation recovery then burns turns retrying, and the run ends with the announcement as finalText without ever executing the tool.

The failure was reproduced client-free: replaying the exact captured request body directly to the provider with curl/a raw script produces the same non-emission 100% of the time (0/10), and adding the tool to the request's tools array fixes it 100% of the time (10/10) with everything else identical.

Expected Behavior

The model should emit a tool_calls block for web_search (and MCP tools) after their schemas have been loaded via search_tools, the harness should execute them, and the run should complete normally — as it does on providers that tolerate undeclared-tool calling.

Actual Behavior
  • Assistant streams reasoning planning the tool call, then streams text announcing it ("Now I'll run the web search for…") and the turn ends with finish_reason: "stop"no tool_calls deltas in the stream.
  • Sometimes the completion is entirely empty (no content, no reasoning) with finish_reason: "stop".
  • The harness fires continuation_recovery (kind: "empty" / kind: "intent") and retries; failures are sticky within a session and the run often ends at max_turns with the announcement as finalText.
  • Only tools whose schemas are declared in tools (e.g. search_tools, shell_command, read_file) work reliably, matching the "simple tools work, web/MCP tools don't" symptom.
  • A/B replay matrix on the captured failing payload (5 repeats per variant, both glm-5.3-flash and glm-5.2):
    • Captured payload as-is → 0/10 tool_calls (text-announce or empty)
    • Same payload with continuation nudge removed → 0/10 (all empty)
    • Same payload + web_search declared in tools10/10 tool_calls
    • Minimal payload (same system+user, only web_search tool) → 10/10 tool_calls

Context size is ruled out: the minimal-payload variant succeeds even though the failing payload carries the full ~96 KB harness context; the deciding factor is solely whether the tool is declared in tools.

Steps to reproduce the issue
  1. Attach any MCP server and use a BYOK provider on the OpenAI-completions wire whose serving enforces declared-tool-only emission (e.g. NeuralWatt glm-5.3-flash or Openrouter glm-5.3-flash).
  2. Run headless with a prompt that requires a withheld tool:
    cmd -p "Search the web for 'z.ai GLM-5.3 flash release notes' and summarize in one sentence." \
      --model <byok-provider>/glm-5.3-flash --output-format json --max-turns 6
    
  3. Observe the NDJSON event stream: turn 1 executes search_tools fine (declared tool), then subsequent turns produce text announcements or empty completions with continuation_recovery events, ending at max_turns without the web search ever running.
  4. Client-free repro (no Command Code involved): replay the captured failing request body directly against the provider endpoint; the same non-emission is deterministic. A ready-made script and trace bundle are available (ab-matrix.mjs + captured request bodies).
Command Code Version

1.40.0

Operating System

Linux

Terminal/IDE

Unknown

Shell

bash

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context
  • Wire evidence was captured with a local logging proxy between Command Code and the provider: full request bodies and raw SSE responses showing finish_reason: "stop" with zero tool_calls deltas despite in-stream reasoning explicitly planning the call. Traces available on request (auth headers redacted).
  • This is a contract gap rather than a one-sided bug: Anthropic-wire endpoints and lenient OpenAI-compatible servers pass tool_calls through without validating against the request's tools array, so progressive disclosure works there by luck. Strict implementations treat tools as the authoritative emission set — a defensible reading of the spec and the safer training contract, since an undeclared tool_calls entry is a hard error downstream. It will reproduce against any strictly-implemented OpenAI-wire provider, not just the one observed here.
  • Suggested fix: for openai-completions-wire providers, declare all available tool schemas up front in tools (or offer a per-provider "declare everything" mode), rather than relying on leniency that only some servers extend. Alternative: a compatibility flag users can set per provider in providers.json.
  • Environment: Command Code CLI 26.7.0 on Linux, BYOK openai-completions wire, streaming with stream_options.include_usage. The client parses tool-call deltas correctly (the declared search_tools call is parsed and executed in every run, including from the same failing session).

貢獻指南

這個儲存庫沒有索引到貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。