anthropics / anthropics/claude-ai-mcp

MCP tool-result images are invisible to the model during the calling turn (Claude Desktop) — model confabulates, then describes correctly on the next user message

Aperta
#643 5 commenti 4 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Nessun dato sulla lingua
Stelle
471
Fork
77
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

## Summary

In Claude Desktop chat, when an MCP tool returns `ImageContent`, the
image renders in the tool-call panel in the UI, but the model's
continuation of the same agentic turn behaves as if the image block is
absent: it reliably "describes" the image from priors (file name,
conversation context, memory). On the **next user message** the model
suddenly sees the image and describes it accurately, often
apologizing for the previous answer. The same server's images are
visible to the model within the calling turn in Claude Code, so this
looks specific to the Desktop/claude.ai chat context assembly.

## Environment

- Claude Desktop 1.22209.0, macOS 15.0.1 (Apple Silicon)
- Chat model: Claude Opus 4.8 (also reproduced with default model)
- MCP server: perception-mcp 0.1.4 → 0.1.7 (PyPI), Python MCP SDK
(FastMCP) 1.28.1, stdio transport, launched via uvx
- Both `image/png` and `image/jpeg` content blocks affected equally

## Repro

Minimal server (20 lines, no dependencies beyond `mcp` + `pillow`):

```python
# repro_server.py
import io, random, string
from mcp.server.fastmcp import FastMCP, Image
from PIL import Image as PILImage, ImageDraw, ImageFont

mcp = FastMCP("image-repro")

@mcp.tool()
def show_secret_word() -> Image:
"""Renders an image containing a secret word. Look at the image
and report the word."""
word = "".join(random.choices(string.ascii_uppercase, k=6))
img = PILImage.new("RGB", (600, 200), "white")
ImageDraw.Draw(img).text((40, 60), word, fill="black",
font=ImageFont.load_default(size=80))
buf = io.BytesIO(); img.save(buf, format="PNG")
return Image(data=buf.getvalue(), format="png")

mcp.run(transport="stdio")
```

1. Add to Claude Desktop, new chat.
2. Ask: **"Call show_secret_word and tell me the word in the image."**
3. Observed: the tool runs, the image with the word renders in the
collapsed tool panel — and the model answers with a **different,
invented word** (or an invented scene description for photographic
images).
4. Reply anything ("are you sure?").
5. Observed: the model now names the word **correctly** and usually
notes its previous answer was wrong.

Step 3 vs step 5 is deterministic across our sessions; with a
random word per call, priors cannot explain the correct second
answer — the image demonstrably reaches the model, one user turn
late.

## Ruled out (server-side hardening that changed nothing)

We shipped three successive result shapes to exclude prompt/attention
causes on our side:

1. `[info-JSON, image]` — model answered from the JSON text.
2. `[info-JSON with explicit "describe only the attached image"
anchor, image]` — still confabulated on the calling turn.
3. **Image alone**, all metadata burned into a caption bar inside the
pixels, zero text blocks — still confabulated on the calling turn
("macro photo of dew drops on a spider web" for a gray noise test
card reading "КЕДР-77" with a blue circle), then described the
card exactly on the next turn.

With (3) there is literally no text to answer from, so the calling
turn's context cannot contain the image. Result size is not the
cause either: all images are ≤ 700 KB encoded (we budget them), well
under the 1 MB tool-result cap.

## Expected

`ImageContent` from an MCP tool result should be part of the model's
context in the same turn that called the tool — as it already is in
Claude Code with the identical server.

## Impact

Any MCP server that returns images for the model to read (charts,
screenshots, rendered pages, OCR aids) silently produces confident
hallucinations in Desktop chat: the model describes what it expects
instead of what the tool returned, and the user has no way to tell.
An honest failure (no image in context → "I can't see it") would
already be an improvement; the actual behavior is worse than the
tool not existing.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start with the minimal repro in repro_server.py and run it through Claude Desktop using the stated steps. Compare the model's response in the calling turn with its response after a follow-up message, and check the same server in Claude Code. Done means ImageContent is available to the model during the original tool-calling turn and the image is described accurately.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
ai, api
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
38/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.