anthropics / anthropics/anthropic-sdk-python

[Beta Sessions / Managed Agents] MCP tool-result image content blocks reach the model as base64 text instead of vision

Abierto
#1,671 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
3.9k
Forks
853
Merge medio
1 d 18 h
PR fusionados (30 d)
11

Descripción

### Summary

In the Beta **Managed Agents / Sessions** API (self-hosted environment — Anthropic runs the agent loop; tools execute in our sandbox), when a configured **remote MCP server** returns a tool result that contains an **image content block** (`{"type":"image","data":"","mimeType":"image/jpeg"}`), the image is **not delivered to the model as vision**. The base64 payload instead reaches the model as **raw text** inside the tool result.

Two consequences:
1. **The model cannot see the image** — it can only work from any accompanying text/OCR block, and explicitly reports it "cannot see" the picture.
2. **Large token waste** — the base64 string is tokenized as text (tens of thousands of tokens per image), with no benefit.

### Expected
MCP tool-result `image` content blocks should be forwarded to the model as image (vision) input — the same way they are in a normal Claude session.

### Actual
The model receives the base64 as text.

### Isolation already done
- The **same MCP server**, called from a normal Claude session (claude.ai / Claude Code via an MCP connector), returns images that render correctly and the model can see them → the server emits valid `ImageContent` blocks.
- For Managed Agents, MCP tools execute **server-side**: the Python SDK's session runner explicitly does **not** dispatch `agent.mcp_tool_use` (in-code comment: *"MCP tools run server-side and the runner never sees a result to post for them"*). So this is in the managed-agents MCP relay, not the local tool runtime.
- `BetaManagedAgentsAgentMcpToolResultEvent`'s content union **already includes `BetaManagedAgentsImageBlock`**, so the type system supports image results — the relay appears to **flatten** `ImageContent` into a text/base64 representation instead of forwarding it as an image block to the model.

### Environment
- `anthropic` (Python) **0.109.1**
- Beta Managed Agents / Sessions, self-hosted environment

### Related
Related to but **distinct from** #1329: that issue is about the **claude.ai UI** not rendering tool images **inline to the user** (where *"Claude can see the image"*). This report is the opposite — in Managed Agents the **model itself does not receive the image** as vision.

### Repro
Configure a managed agent with a remote MCP server whose tool returns `[TextContent, ImageContent]`; run a session that calls that tool; observe that the model receives the base64 as text rather than seeing the image. Happy to provide a minimal MCP server and a session id on request.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.