posit-dev / posit-dev/chatlas

Tracking: expand first-class built-in (provider-executed) tools

Open
#369 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-triage:needs-review
Dominant language
Python
Stars
176
Forks
28
Avg merge
18h 42m
Merged PRs (30d)
16

Description

chatlas currently ships two first-class built-in (provider-executed) tools: tool_web_search() and tool_web_fetch() (#248), plus the generic ToolBuiltIn escape hatch (#134). This is a tracking issue for expanding that lineup, based on a July 2026 survey of the provider-executed tools offered by Anthropic (Messages API), OpenAI (Responses API), and Google (Gemini API / Vertex AI).

What the survey found

The three first-class providers have converged on a similar set of server-executed tools — the kind that fit the existing ToolBuiltIn model, where the provider runs the tool and chatlas only needs to send a config and parse result content:

Tool Anthropic OpenAI (Responses) Google Gemini
Web search ✅ GA (shipped) ✅ GA (shipped) ✅ GA (shipped)
Web fetch / URL context ✅ GA (shipped) ✅ GA (shipped)
Code execution ✅ GA ✅ GA (code_interpreter) ✅ GA
Remote MCP connector ✅ beta ✅ GA (mcp) ⚠️ experimental (not Vertex)
File search (hosted RAG) ✅ GA ✅ GA (not Vertex)
Tool search / deferred tools ✅ GA ✅ GA
Image generation ✅ GA ❌ (separate models)
Maps grounding ✅ GA

A second family is provider-defined but client-executed (Anthropic: memory / bash / text editor / computer use; OpenAI: computer use / apply_patch / local shell; Google: computer use). These have official schemas but the application must execute them, so they don't fit ToolBuiltIn — supporting them means shipping an execution harness, a much larger design.

Sub-issues

Roughly by priority:

  • #364 — tool_code_execution(): the only remaining server tool that is GA across all three providers
  • #365 — server-side MCP connector: complements the existing client-side register_mcp_tools_* support
  • #367 — make built-in tools fail loudly on providers/platforms that don't support them (pre-existing bug; grows in importance with each new tool)
  • #366 — (decision) file search / provider-hosted RAG: in tension with the "no RAG primitives" non-goal from #360; may close as "no"

Explicitly not planned (for now)

  • Computer use, bash, text editor, apply-patch, shell — client-executed; chatlas would have to ship a screenshot/action or command-execution harness. Out of scope for a chat client.
  • Image generation (OpenAI-only) and Maps grounding (Gemini-only) — single-provider; the ToolBuiltIn escape hatch already handles them (image generation is the worked example in #134). A docs recipe may be worth more than first-class wrappers.
  • Anthropic tool search / advisor tool — too new/niche; revisit if users accumulate large deferred-tool catalogs.
  • Anthropic memory tool — client-executed but cheap to back with a filesystem; interesting, but a different pattern from ToolBuiltIn. Revisit separately if there's demand.

Cross-provider caveats that affect all of the above

  • Amazon Bedrock supports essentially none of Anthropic's server tools; Vertex AI supports only basic Anthropic web search. #367 covers surfacing this early.
  • Gemini models before Gemini 3 can't mix built-in tools with custom function declarations in one request.
  • ellmer currently has no code execution or structured citation model, so this is an area where chatlas leads; worth coordinating naming/design with the ellmer team as these land.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing ToolBuiltIn model and the linked sub-issues #364, #365, #367, and #366 to understand the proposed scope and provider constraints. This tracking issue is done when the selected sub-issues are resolved or explicitly closed with a documented decision; it does not identify a specific file or test to change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.