Tracking: expand first-class built-in (provider-executed) tools
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 176
- Forks
- 28
- Avg merge
- 18h 42m
- Merged PRs (30d)
- 16
Description
chatlas currently ships two first-class built-in (provider-executed) tools: tool_web_search() and tool_web_fetch() (#248), plus the generic ToolBuiltIn escape hatch (#134). This is a tracking issue for expanding that lineup, based on a July 2026 survey of the provider-executed tools offered by Anthropic (Messages API), OpenAI (Responses API), and Google (Gemini API / Vertex AI).
What the survey found
The three first-class providers have converged on a similar set of server-executed tools — the kind that fit the existing ToolBuiltIn model, where the provider runs the tool and chatlas only needs to send a config and parse result content:
| Tool | Anthropic | OpenAI (Responses) | Google Gemini |
|---|---|---|---|
| Web search | ✅ GA (shipped) | ✅ GA (shipped) | ✅ GA (shipped) |
| Web fetch / URL context | ✅ GA (shipped) | ❌ | ✅ GA (shipped) |
| Code execution | ✅ GA | ✅ GA (code_interpreter) |
✅ GA |
| Remote MCP connector | ✅ beta | ✅ GA (mcp) |
⚠️ experimental (not Vertex) |
| File search (hosted RAG) | ❌ | ✅ GA | ✅ GA (not Vertex) |
| Tool search / deferred tools | ✅ GA | ✅ GA | ❌ |
| Image generation | ❌ | ✅ GA | ❌ (separate models) |
| Maps grounding | ❌ | ❌ | ✅ GA |
A second family is provider-defined but client-executed (Anthropic: memory / bash / text editor / computer use; OpenAI: computer use / apply_patch / local shell; Google: computer use). These have official schemas but the application must execute them, so they don't fit ToolBuiltIn — supporting them means shipping an execution harness, a much larger design.
Sub-issues
Roughly by priority:
- #364 —
tool_code_execution(): the only remaining server tool that is GA across all three providers - #365 — server-side MCP connector: complements the existing client-side
register_mcp_tools_*support - #367 — make built-in tools fail loudly on providers/platforms that don't support them (pre-existing bug; grows in importance with each new tool)
- #366 — (decision) file search / provider-hosted RAG: in tension with the "no RAG primitives" non-goal from #360; may close as "no"
Explicitly not planned (for now)
- Computer use, bash, text editor, apply-patch, shell — client-executed; chatlas would have to ship a screenshot/action or command-execution harness. Out of scope for a chat client.
- Image generation (OpenAI-only) and Maps grounding (Gemini-only) — single-provider; the
ToolBuiltInescape hatch already handles them (image generation is the worked example in #134). A docs recipe may be worth more than first-class wrappers. - Anthropic tool search / advisor tool — too new/niche; revisit if users accumulate large deferred-tool catalogs.
- Anthropic memory tool — client-executed but cheap to back with a filesystem; interesting, but a different pattern from
ToolBuiltIn. Revisit separately if there's demand.
Cross-provider caveats that affect all of the above
- Amazon Bedrock supports essentially none of Anthropic's server tools; Vertex AI supports only basic Anthropic web search. #367 covers surfacing this early.
- Gemini models before Gemini 3 can't mix built-in tools with custom function declarations in one request.
- ellmer currently has no code execution or structured citation model, so this is an area where chatlas leads; worth coordinating naming/design with the ellmer team as these land.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing ToolBuiltIn model and the linked sub-issues #364, #365, #367, and #366 to understand the proposed scope and provider constraints. This tracking issue is done when the selected sub-issues are resolved or explicitly closed with a documented decision; it does not identify a specific file or test to change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100