[FEATURE] Cross-provider tool search / deferred tool loading (design proposal)

Scope check

• This is core LLM communication (not application logic)
• This benefits most users (not just my use case)
• This can't be solved in application code with current RubyLLM
• I read the Contributing Guide

Due diligence

• I searched existing issues
• I checked the documentation

What problem does this solve?

Written by my own hand: I used AI to assist me with this issue, but I've worked hard to ensure it is as concise and accurate as possible, without losing any fidelity.

A provider-agnostic way to mark tools as deferred so their schemas stay out of the model's context until the model discovers them on demand via whatever tool-search mechanism the underlying provider offers. This is the design discussion requested in #745, and it supersedes the Anthropic-specific framing of #748.

Background

Two prior threads set this up:

• #748 — [FEATURE] Support Anthropic Tool Search (deferred tool loading) (by @skovy). A thorough request, but scoped to Anthropic's tool-search tool specifically.
• #745 — Translate Anthropic tool search feature (defer_loading + native BM25) (by @swistaczek). A complete, tested implementation that @crmne closed with (paraphrasing): the problem is real, but the API shape was too Anthropic-specific and added a lot of public surface before the abstraction was agreed on — and, notably, OpenAI now has native tool search too — so this "wants to be a RubyLLM-level tool search / tool loading feature with provider adapters underneath, not an Anthropic-only API," with a request to open an issue for the design first.

This issue is that design proposal. Thanks to both @skovy and @swistaczek for the groundwork.

The problem

When a Chat is wired to many tools — especially across one or more MCP servers — every tool's full JSON Schema ships on every request. Three costs follow:

1. Token bloat. Hundreds of tools add tens of thousands of tokens per request.
2. Prompt-cache eviction. Changing the tool set changes the request prefix and invalidates the cache.
3. Selection accuracy. Models pick worse tools as the menu grows past a few dozen.

Providers have started solving this natively, and — importantly for RubyLLM — they've converged: Anthropic and OpenAI independently landed on the same defer_loading: true flag plus a "search tool" the model calls to pull in the tools it needs. That convergence is the strongest signal that this belongs behind one RubyLLM abstraction rather than a per-provider API.

Provider landscape (mid-2026)

Provider	Native support	Shape
Anthropic	Yes (GA; 4.5+ and 5.x models)	defer_loading: true on tools + a tool_search_tool_{regex,bm25}_* primitive; discovered tools come back as server_tool_use + tool_search_tool_result blocks, which the API expects replayed in later requests
OpenAI — Responses API	Yes (gpt-5.4+)	defer_loading: true + a {"type": "tool_search"} tool; results as tool_search_call / tool_search_output items; discovered-tool function_call items carry a namespace field that must be round-tripped (verified live — the API 400s without it, and this is easy to miss from the docs)
OpenAI — Chat Completions	No	—
AWS Bedrock	Anthropic's, but InvokeModel only (not the Converse API RubyLLM speaks)	same blocks as Anthropic
Google Gemini	No native discovery	function-calling config can constrain the callable set, but there's no on-demand search
Mistral / DeepSeek / Ollama / Perplexity / OpenRouter / GPUStack	No	standard function calling only

There is no ratified cross-provider standard (MCP's tool-filtering proposal, SEP-1300, is not merged), but the Anthropic/OpenAI convergence gives a stable core to model against, with a graceful fallback for everyone else.

Proposed design (overview)

The guiding principle from #745's review: a RubyLLM-level feature with provider adapters underneath. Keep the public surface small and provider-neutral; push every wire-format detail into the protocol layer.

RubyLLM-level (provider-agnostic)

• Tool.deferred — a class-level DSL marking a tool deferrable, plus a per-call Chat#with_tools(*tools, defer: true/false) override.
• Chat#tool_catalog — a ToolCatalog holding the deferred tools, separate from the active tool set.
• Message#tool_references — the normalized channel by which a provider reports "these tools were discovered."
• Chat#after_tool_search — a callback receiving the newly discovered tool names (an Array of Symbols), fired idempotently.

Three design decisions worth calling out

1. The tools array is identical on every request. Deferred tools stay deferred on the wire even after the model discovers them, and discovery never mutates the rendered tool set — so the provider's prompt cache survives the whole conversation. (This is where a naive "promote discovered tools to active" design goes wrong: it rewrites the payload on exactly the turn after every discovery, busting the cache the feature exists to protect. Measured live on Anthropic: after the first turn, per-turn uncached input drops to single-digit tokens with the entire prefix riding the cache.)

2. The provider's search exchange is replayed in history. The raw provider-native search blocks are kept on the message and replayed verbatim in later requests (Anthropic requires complete server_tool_use + result pairs — replaying half is a 400, verified live; omitting a whole pair merely costs a re-search, also verified). Replay is gated on the current request still carrying deferred tools, and type-filtered per provider, so switching models mid-chat — including automatic fallbacks, cross-provider ones included — degrades transparently instead of poisoning requests with foreign block types.

3. Deferral intent is resolved at render time, per request. Registration records intent; every request re-checks the current provider/model (via a provider Capabilities.supports_tool_search?(model_id) predicate, following the existing capabilities pattern). Unsupported provider/model ⇒ the catalog is sent as ordinary eager tools with a one-time warning. Model switches and fallbacks transparently activate or degrade deferral — the same application code runs everywhere. Dispatch falls back to the catalog, so a discovered tool executes exactly like an active one (deferral is a context optimization, not an authorization boundary).

The provider-adapter seam

A single capability hook — Protocol#supports_deferred_tools? (default false, model-aware) — plus two responsibilities each supporting protocol implements privately:

1. Render: emit the provider's defer flag on deferred tools and append its native search primitive.
2. Parse/replay: turn the provider's discovery output into Message#tool_references + raw replayable blocks.

Nothing else crosses the seam. The generic layer never sees tool_search_tool_bm25, tool_search, namespace, or tool_search_output — each adapter owns its own vocabulary.

Additive by construction

If you don't use defer: or deferred, nothing changes: the rendered payload is byte-identical to today's.

Reference implementation

A complete implementation exists on a fork branch — not opened as a PR, pending agreement on the shape here:

• Diff vs. main: https://github.com/crmne/ruby_llm/compare/main...benjaminwood:ruby_llm:tool-search

[Update] on top of 2.0: https://github.com/crmne/ruby_llm/compare/v2.0.0...benjaminwood:ruby_llm:tool-search-2

State of that branch, for calibration:

• Both adapters (Anthropic + OpenAI Responses) verified against the live APIs, streaming and non-streaming, multi-turn with cross-turn reuse of discovered tools. Live-measured on Anthropic: a 17-tool catalog deferred cuts first-request input tokens ~50%, and with prompt caching the tools-array SHA is identical across every request of a conversation.
• Rails/acts_as persistence follows the existing optional-column (has_attribute?) pattern: with tool_references/tool_search_blocks columns the search exchange survives restarts; without them everything still works and the model just re-searches.
• Full suite green (1968 examples), RuboCop clean; docs page included.
• It also survived several rounds of deliberately adversarial review; the fixes that came out of that (model-aware capability gating incl. date-suffixed snapshot ids, cross-provider replay filtering, the OpenAI namespace round-trip) are in the branch.

Deliberately left for follow-ups (not in the v1 surface)

• Search-variant selection (Anthropic regex vs. BM25) — the impl defaults to BM25; a knob is additive.
• Custom / client-side search (returning tool references from your own embeddings-based tool, per #748's item 4). The discovery/replay machinery is deliberately decoupled from how references are produced, so this slots in later.
• Bedrock via InvokeModel — RubyLLM's Bedrock provider speaks Converse, which can't do tool search; supporting it is an orthogonal transport question.
• OpenAI namespace grouping of tools — useful, provider-specific, additive.

Prior art

• Supersedes #748 (Anthropic-specific request, @skovy).
• Builds on the closed #745 (@swistaczek) and @crmne's review direction on it.

Why this belongs in RubyLLM

The need has been acknowledged by @crmne and he asked for an issue to discuss the solution.

Happy to reshape any of this per the discussion, then open a PR against the agreed surface.

Start by reviewing the proposed Tool.deferred, Chat#tool_catalog, Message#tool_references, and Chat#after_tool_search entry points, along with the linked fork branch and its provider-adapter approach. This is design work: done means the provider-neutral surface and adapter responsibilities are agreed before a pull request is opened.