github / github/app

Tool Search never defers MCP tools in the desktop app — gated behind an unassigned server experiment

Open
#3,172 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
No language data
Stars
2.1k
Forks
153
PR merge metrics
No merged PRs in 30d

Description

Investigated on macOS (App engine `1.0.80`, model `claude-opus-4.8`). MCP tool definitions load eagerly in every session regardless of model or config.

Measured (via `~/.copilot/data.db` → `sessions.context_tool_definitions_tokens`, on a copy): a fresh session whose only message was `hi` = 151,111 tool-def tokens. Same value across `claude-opus-4.8`, and up to 406k–440k on `gpt-5.6-sol` / `claude-opus-5` — so this is not model gating; all are "supported" models per the docs.

Root cause: deferral (`Session::apply_tool_search_defer_policy`, event `tool_search.activated`, present in the app binary) is gated behind the server-assigned `tool_search` experiment flight. My `app_state → exp-assignment-cache` (FlightingVersion 5389) only assigns `ccr_pr_nudge_adoption` + `app_aa_c1`; `tool_search` is absent. In the binary, `tool_search` sits in the same flight family as `mcp_apps` / `local_sandbox` / `cloud_sessions` — i.e. a server rollout, not a user toggle (which is why it also doesn't appear in Settings → Experimental).

Things that did NOT work:

* `deferTools: "auto"` per server in `mcp-config.json`
* Hard-adding `tool_search` to `app_state → enabled-experiments` in `data.db`
* `toolSearch: true` in `~/.copilot/settings.json`, tested both before and after a full app restart → tool-def tokens unchanged (151,111 → 151,115). The app constructs `toolSearch` itself from `expAssignments` and ignores this settings.json key, unlike the documented CLI.

Ask: please expose Tool Search as a user setting in the desktop app that bypasses the experiment gate, or enroll accounts in the `tool_search` flight — otherwise heavy MCP users pay for 150k–440k tool-definition tokens on every turn with no local mitigation.

---

Related: #1398 (dynamic MCP tool loading request), #2141 (large tool sets silently produce no output on Claude models in the app, while the CLI — where tool search is on by default — is unaffected). CLI docs: https://docs.github.com/en/copilot/concepts/agents/copilot-cli/tool-search

Contributor guide

Open the contributing guide

Research direction

Start by examining the desktop binary's Session::apply_tool_search_defer_policy entry point, the tool_search.activated event, and the expAssignments flight data described in app_state → exp-assignment-cache. Compare this behavior with the CLI tool-search documentation and the documented settings paths. Done means the desktop app either offers a user-controlled bypass or enrolls accounts in tool_search, with the reported per-session tool-definition token count reduced.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos
Domain
ai-infra-agents, desktop
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.