MCP tool routing divergence: deferred MCP tools fail "unknown tool" 1ms on direct invoke (search-mode bug)

Symptom

Same MCP server (gdrive), same agent (Leader), 3 tools in 3 different states:

A second agent (Director) on the same MCP server can invoke all 3 tools directly.

The 3-orders-of-magnitude timing gap (1-5ms vs 583-1002ms) proves the failure happens at the tenant/registry router level, before any request reaches the gdrive MCP server.

Root cause investigation — TWO hypotheses

The actual root cause depends on whether gdrive has settings.require_user_credentials = true. The two paths through the code behave differently.

Hypothesis A — gdrive is NOT user-cred (standard path)

If gdrive goes through LoadForAgent → connectViaPool → maybeEnterSearchMode, the bug is in two architectural defects:

A1. Non-deterministic inline/deferred split in internal/mcp/manager.go:359-417:

func (m *Manager) maybeEnterSearchMode() {
    allNames := m.ToolNames()  // iterates m.servers (Go map = random order)
    if len(allNames) <= mcpToolInlineMaxCount { return }
    deferSet := ...
    for _, name := range allNames[mcpToolInlineMaxCount:] { // arbitrary slice
        deferSet[name] = struct{}{}
    }
}

Each Manager construction produces a different inline-vs-deferred split because Go map iteration is undefined-order. Two agents with identical grants disagree on which tools are directly callable. Threshold = 40 tools (mcpToolInlineMaxCount constant in same file, line 31). Director likely has < 40 tools total → no search mode → all-inline → all 3 tools work direct.

A2. Lazy activator wired but never called from execution path. internal/agent/resolver.go:329-330:

// Search mode: too many tools — register mcp_tool_search meta-tool.
// Also wire lazy activator so deferred tools can be called by name directly.
toolsReg.SetDeferredActivator(mcpMgr.ActivateToolIfDeferred)

The comment promises "deferred tools can be called by name directly," but Registry.ExecuteWithContext (internal/tools/registry.go:176-208) never invokes TryActivateDeferred — only test code (loop_lazy_mcp_test.go) does. Result: deferred tools require an explicit mcp_tool_search round-trip before they can be invoked, despite the registry hook being wired correctly.

Hypothesis B — gdrive IS user-cred (per-request path)

If gdrive has require_user_credentials: true, tools go through getUserMCPTools (internal/agent/loop_mcp_user.go) which registers them directly into the registry per-request. In this path, tools NEVER enter m.deferredTools — but mcp_tool_search builds its BM25 index only from m.deferredTools (internal/mcp/mcp_tool_search.go:32). So the visibility evidence (search returns getGoogleSheetContent) would be impossible if gdrive is user-cred.

If hypothesis B is correct AND search-visibility evidence stands, then either (a) the user misread search results (different tool/server with similar name), or (b) there's a separate bug in getUserMCPTools we have not located yet — candidates:

• Cache returns stale list, registry has dropped tools (loop_mcp_user.go:24-37): cache check only verifies IsConnected(), not whether tools are still in the registry.
• Skip-if-exists registration (loop_mcp_user.go:95-99): name collision with another server skips registration but still appends to user's cache.
• Pool entry caches tools at first connect: pool never re-discovers if MCP server adds tools later.

Verification needed

SELECT name, settings->'require_user_credentials' AS user_cred
FROM mcp_servers
WHERE id = '019da177-3d1d-76d1-b2ab-e9bb2af0fd00';

Or check gateway logs when Leader resolves:

• mcp.server.deferred_user_creds server=gdrive → user-cred path (hypothesis B)
• mcp.search_mode.enabled inline_tools=40 deferred_tools=N (with agent=Leader) → standard path (hypothesis A)

Proposed fix (covers hypothesis A)

Branch: fix/mcp-deferred-tool-direct-invoke (local, not yet pushed)

Layer 1 — sort allNames lexicographically before slicing in maybeEnterSearchMode. Makes the inline/deferred split deterministic across agents and reloads.

Layer 2 — wire TryActivateDeferred into Registry.ExecuteWithContext so deferred MCP tools can be invoked directly without an explicit mcp_tool_search first.

Total: ~20 LOC production change + 3 new tests. Builds clean (PG + sqliteonly). All registry/manager/lazy-MCP tests pass. Hot path (tool already registered) takes 0 extra instructions.

Workarounds (until fix lands)

1. Have the LLM call mcp_tool_search with keyword matching the desired tool name before invoking. After search returns, the tool is activated and callable directly. (Behavior already documented in mcp_tool_search description.)
2. Reduce per-agent grant count to keep total MCP tools under 40 → no search mode → all-inline → all tools directly callable.
3. Toggle the grant via UI to force agentRouter.InvalidateAll() and re-roll the map iteration order. Unreliable — Go map order is not strictly random.

Notes for fix author

• uploadFile NOT VISIBLE in search is not part of this bug — likely filtered by tool_allow/tool_deny grant settings or simply scored too low by BM25 for the query keywords used. Verify grant config before chasing.
• mcp_tool_search.rebuildIndex() is called only at construction (mcp_tool_search.go:27) and never refreshed when deferredTools changes. Becomes a non-issue once Layer 2 lands (direct invoke works regardless of search index freshness), but worth noting.
• Pool entry's discovered tool list is cached at first connectAndDiscover and never re-discovered (see comment at manager_connect.go:380). If MCP server adds tools later, goclaw will not see them until the pool entry is evicted and recreated. Out of scope for this issue.

First verify the gdrive credential setting with the provided SQL or gateway logs to identify the relevant path. Then inspect internal/mcp/manager.go, internal/agent/resolver.go, internal/tools/registry.go, and the user-MCP files, starting with the existing lazy-MCP tests. Done means the inline/deferred assignment is deterministic and deferred tools can be directly invoked, with the proposed registry and manager tests passing.