anthropics / anthropics/anthropic-sdk-python

[Beta Sessions / Managed Agents] MCP tool-result image content blocks reach the model as base64 text instead of vision

Offen
#1,671 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
3.9k
Forks
853
Ø Merge
1 T. 18 Std.
Gemergte PRs (30 T.)
11

Beschreibung

### Summary

In the Beta **Managed Agents / Sessions** API (self-hosted environment — Anthropic runs the agent loop; tools execute in our sandbox), when a configured **remote MCP server** returns a tool result that contains an **image content block** (`{"type":"image","data":"","mimeType":"image/jpeg"}`), the image is **not delivered to the model as vision**. The base64 payload instead reaches the model as **raw text** inside the tool result.

Two consequences:
1. **The model cannot see the image** — it can only work from any accompanying text/OCR block, and explicitly reports it "cannot see" the picture.
2. **Large token waste** — the base64 string is tokenized as text (tens of thousands of tokens per image), with no benefit.

### Expected
MCP tool-result `image` content blocks should be forwarded to the model as image (vision) input — the same way they are in a normal Claude session.

### Actual
The model receives the base64 as text.

### Isolation already done
- The **same MCP server**, called from a normal Claude session (claude.ai / Claude Code via an MCP connector), returns images that render correctly and the model can see them → the server emits valid `ImageContent` blocks.
- For Managed Agents, MCP tools execute **server-side**: the Python SDK's session runner explicitly does **not** dispatch `agent.mcp_tool_use` (in-code comment: *"MCP tools run server-side and the runner never sees a result to post for them"*). So this is in the managed-agents MCP relay, not the local tool runtime.
- `BetaManagedAgentsAgentMcpToolResultEvent`'s content union **already includes `BetaManagedAgentsImageBlock`**, so the type system supports image results — the relay appears to **flatten** `ImageContent` into a text/base64 representation instead of forwarding it as an image block to the model.

### Environment
- `anthropic` (Python) **0.109.1**
- Beta Managed Agents / Sessions, self-hosted environment

### Related
Related to but **distinct from** #1329: that issue is about the **claude.ai UI** not rendering tool images **inline to the user** (where *"Claude can see the image"*). This report is the opposite — in Managed Agents the **model itself does not receive the image** as vision.

### Repro
Configure a managed agent with a remote MCP server whose tool returns `[TextContent, ImageContent]`; run a session that calls that tool; observe that the model receives the base64 as text rather than seeing the image. Happy to provide a minimal MCP server and a session id on request.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.