CommandCodeAI / CommandCodeAI/command-code

Tool calls silently fail on strict OpenAI-wire BYOK providers: web_search/MCP tools announced in text or empty completion instead of tool_use — undeclared-tool progressive disclosure not tolerated by all servers

未关闭
#787 1 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

@ahmadbilaldev 已经在做这个了。

开始于 2026年9月17日。

主要语言
没有语言数据
星标
4k
派生
350
PR 合并指标
30 天内没有已合并 PR

描述

Summary

Command Code uses progressive tool disclosure: most tool schemas (web_search, web_fetch, all mcp__* tools) are not declared in the request's tools array. Their schemas are delivered lazily as the text result of a search_tools call, and the model is expected to emit a tool_calls block for a tool that is not present in the OpenAI tools parameter of that request.

Some OpenAI-wire serving stacks tolerate this (emitting tool_calls for undeclared, in-prompt-described tools); others — in our case NeuralWatt's GLM serving (glm-5.3-flash, glm-5.2) and Openrouter — strictly constrain emission to declared tools. On those servers the model either writes a text reply announcing the call ("Now I'll run the web search…") with finish_reason: "stop" and zero tool_calls deltas, or returns a completely empty completion. The harness's continuation recovery then burns turns retrying, and the run ends with the announcement as finalText without ever executing the tool.

The failure was reproduced client-free: replaying the exact captured request body directly to the provider with curl/a raw script produces the same non-emission 100% of the time (0/10), and adding the tool to the request's tools array fixes it 100% of the time (10/10) with everything else identical.

Expected Behavior

The model should emit a tool_calls block for web_search (and MCP tools) after their schemas have been loaded via search_tools, the harness should execute them, and the run should complete normally — as it does on providers that tolerate undeclared-tool calling.

Actual Behavior
  • Assistant streams reasoning planning the tool call, then streams text announcing it ("Now I'll run the web search for…") and the turn ends with finish_reason: "stop"no tool_calls deltas in the stream.
  • Sometimes the completion is entirely empty (no content, no reasoning) with finish_reason: "stop".
  • The harness fires continuation_recovery (kind: "empty" / kind: "intent") and retries; failures are sticky within a session and the run often ends at max_turns with the announcement as finalText.
  • Only tools whose schemas are declared in tools (e.g. search_tools, shell_command, read_file) work reliably, matching the "simple tools work, web/MCP tools don't" symptom.
  • A/B replay matrix on the captured failing payload (5 repeats per variant, both glm-5.3-flash and glm-5.2):
    • Captured payload as-is → 0/10 tool_calls (text-announce or empty)
    • Same payload with continuation nudge removed → 0/10 (all empty)
    • Same payload + web_search declared in tools10/10 tool_calls
    • Minimal payload (same system+user, only web_search tool) → 10/10 tool_calls

Context size is ruled out: the minimal-payload variant succeeds even though the failing payload carries the full ~96 KB harness context; the deciding factor is solely whether the tool is declared in tools.

Steps to reproduce the issue
  1. Attach any MCP server and use a BYOK provider on the OpenAI-completions wire whose serving enforces declared-tool-only emission (e.g. NeuralWatt glm-5.3-flash or Openrouter glm-5.3-flash).
  2. Run headless with a prompt that requires a withheld tool:
    cmd -p "Search the web for 'z.ai GLM-5.3 flash release notes' and summarize in one sentence." \
      --model <byok-provider>/glm-5.3-flash --output-format json --max-turns 6
    
  3. Observe the NDJSON event stream: turn 1 executes search_tools fine (declared tool), then subsequent turns produce text announcements or empty completions with continuation_recovery events, ending at max_turns without the web search ever running.
  4. Client-free repro (no Command Code involved): replay the captured failing request body directly against the provider endpoint; the same non-emission is deterministic. A ready-made script and trace bundle are available (ab-matrix.mjs + captured request bodies).
Command Code Version

1.40.0

Operating System

Linux

Terminal/IDE

Unknown

Shell

bash

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context
  • Wire evidence was captured with a local logging proxy between Command Code and the provider: full request bodies and raw SSE responses showing finish_reason: "stop" with zero tool_calls deltas despite in-stream reasoning explicitly planning the call. Traces available on request (auth headers redacted).
  • This is a contract gap rather than a one-sided bug: Anthropic-wire endpoints and lenient OpenAI-compatible servers pass tool_calls through without validating against the request's tools array, so progressive disclosure works there by luck. Strict implementations treat tools as the authoritative emission set — a defensible reading of the spec and the safer training contract, since an undeclared tool_calls entry is a hard error downstream. It will reproduce against any strictly-implemented OpenAI-wire provider, not just the one observed here.
  • Suggested fix: for openai-completions-wire providers, declare all available tool schemas up front in tools (or offer a per-provider "declare everything" mode), rather than relying on leniency that only some servers extend. Alternative: a compatibility flag users can set per provider in providers.json.
  • Environment: Command Code CLI 26.7.0 on Linux, BYOK openai-completions wire, streaming with stream_options.include_usage. The client parses tool-call deltas correctly (the declared search_tools call is parsed and executed in every run, including from the same failing session).

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。