Tool search reports enabled on Grok but defers nothing (57.7k vs 21.0k tokens); Gemini never enables it — follow-up to #4588

Summary

Follow-up to #4588 (closed as completed on 2026-08-26).

That fix landed for GPT — gpt-5.6-sol went from 43.1k → 21.0k tokens on a one-word prompt and now defers 140/160 tools, making it the cheapest model. But two families were left behind, and they fail in two different ways:

• Grok reports tool_search_enabled: true and generic_flag_service_available: Some(true) — yet deferred_tool_count: 0 and every tool token is still billed. Tool search is enabled and then not applied.
• Gemini never enables it at all: all three flag paths None, tool_search_enabled: false, deferred: 0.

Grok at 57.7k tokens for the prompt "hi" is now 2.75× the cost of gpt-5.6-sol at 21.0k, on an identical machine, config, and MCP set.

The Grok case is the more serious of the two, because the feature reports itself as ON. Any dashboard or check keying off tool_search_enabled will show Grok as healthy while it defers nothing.

Environment

• Copilot CLI 1.0.82-0, macOS (darwin-arm64)
• 10 MCP servers, ~161 tools
• ~/.copilot/settings.json: toolSearch: true, experimental: true

Results

Identical machine / config / MCP set. Prompt is the single word hi. Every row reproduced across 3 independent runs with <10 token variance.

Model	Input tokens	Tools	Deferred	tool_search_enabled	flag service	Verdict
gpt-5.6-sol	20,969	160	140	false	all None	✅ defers
claude-opus-5	33,346	162	140	false	builtin=Some(true)	✅ defers
gemini-3.7-flash	46,776	161	0	false	all None	❌ never enabled
gemini-3.1-pro-preview	46,778	161	0	false	all None	❌ never enabled
grok-4.5	57,667	161	0	true	generic=Some(true)	❌ enabled, not applied
grok-4.6	57,671	161	0	true	generic=Some(true)	❌ enabled, not applied

Note the inverted signal: the two models that say tool search is enabled (Grok) defer nothing, while the two that say false (GPT, Claude) defer 140 tools each. Whatever tool_search_enabled currently reflects, it is not whether deferral is applied.

How to reproduce — no extra tooling

Method 1 — the CLI already prints it. The end-of-run banner reports input tokens directly, so a bare one-liner is enough:

for m in gpt-5.6-sol claude-opus-5 gemini-3.7-flash grok-4.6; do
  echo -n "$m -> "
  copilot -p "hi" --model "$m" --no-custom-instructions --stream off 2>&1 | grep -iE '^Tokens'
done

Actual output:

gpt-5.6-sol      -> Tokens ↑ 21.0k (21.0k written) • ↓ 11
claude-opus-5    -> Tokens ↑ 33.3k (18.8k cached, 14.5k written) • ↓ 13
gemini-3.7-flash -> Tokens ↑ 46.8k • ↓ 9 (20 reasoning)
grok-4.6         -> Tokens ↑ 57.7k • ↓ 35 (26 reasoning)

Method 2 — interactively. In a session, /model grok-4.6, send hi, then /usage for billed input tokens. /context shows the context-window breakdown, though per #4189 it reports the un-deferred footprint and so will not reflect savings; /usage is the trustworthy number.

Method 3 — the deciding debug field. turn tool surface resolved, emitted by copilot_runtime::session::native_message_turn:

m=grok-4.6
copilot -p "hi" --model $m --no-custom-instructions --stream off \
  --log-level debug --log-dir /tmp/ts-$m >/dev/null 2>&1
grep -ho 'turn tool surface resolved {.*}' /tmp/ts-$m/*.log | head -1 \
  | sed 's/turn tool surface resolved //' \
  | jq -c '{tools:.tool_count, deferred:.deferred_tool_count, ts:.tool_search_enabled,
            b:.builtin_flag_service_available, g:.generic_flag_service_available}'

// grok-4.6 — enabled, but nothing deferred
{"tools":161,"deferred":0,"ts":true,"b":"None","g":"Some(true)"}

// gpt-5.6-sol — not "enabled", yet defers 140
{"tools":160,"deferred":140,"ts":false,"b":"None","g":"None"}

Note: the repro script in #4588 piped grep output straight into jq, which fails — the match still carries the turn tool surface resolved prefix. The sed above fixes it.

Additional observation

Independent of which model defers, the client appears to send the full tool catalogue on every turn — roughly 48–49k tokens' worth of schemas for all six models above, near-identical in size. For GPT and Claude most of it is evidently discarded before it reaches the provider; for Grok none of it is. That points at the pruning being applied downstream of the client rather than by it, which would explain why a client-visible flag can read true while having no effect. Possibly related to #4613.

Questions

1. For Grok: tool_search_enabled: true with deferred_tool_count: 0 — is the generic flag path wired to anything that actually prunes, or is it currently reporting-only?
2. Is Gemini enablement planned, or deliberately excluded?
3. Should tool_search_enabled be treated as authoritative? Right now deferred_tool_count is the only field that reflects reality.

Related

• #4588 — original report, closed as completed (fixed for GPT only)
• #4613 — MCP schemas eagerly injected, +354K startup tokens
• #4189 — /context reports un-deferred footprint
• #2901 — lazy-load MCP servers

Tái hiện phép so sánh model bằng các lệnh CLI được cung cấp, sau đó kiểm tra output turn tool surface resolved từ copilot_runtime::session::native_message_turn. So sánh tool_search_enabled với deferred_tool_count cho Grok, Gemini, GPT và Claude; được xem là hoàn thành khi hành vi bật và tín hiệu trạng thái có thẩm quyền đã được giải thích, bao gồm cả việc Gemini có bị loại trừ một cách có chủ đích hay không.