anthropics / anthropics/claude-ai-mcp

MCP tool-result images are invisible to the model during the calling turn (Claude Desktop) — model confabulates, then describes correctly on the next user message

オープン
#643 コメント 5 件 リアクション 4 件 担当者 0 名 GitHub で見る
主要言語
言語のデータがありません
スター
471
フォーク
76
PR マージ指標
30日以内にマージされた PR はありません

説明

## Summary

In Claude Desktop chat, when an MCP tool returns `ImageContent`, the
image renders in the tool-call panel in the UI, but the model's
continuation of the same agentic turn behaves as if the image block is
absent: it reliably "describes" the image from priors (file name,
conversation context, memory). On the **next user message** the model
suddenly sees the image and describes it accurately, often
apologizing for the previous answer. The same server's images are
visible to the model within the calling turn in Claude Code, so this
looks specific to the Desktop/claude.ai chat context assembly.

## Environment

- Claude Desktop 1.22209.0, macOS 15.0.1 (Apple Silicon)
- Chat model: Claude Opus 4.8 (also reproduced with default model)
- MCP server: perception-mcp 0.1.4 → 0.1.7 (PyPI), Python MCP SDK
(FastMCP) 1.28.1, stdio transport, launched via uvx
- Both `image/png` and `image/jpeg` content blocks affected equally

## Repro

Minimal server (20 lines, no dependencies beyond `mcp` + `pillow`):

```python
# repro_server.py
import io, random, string
from mcp.server.fastmcp import FastMCP, Image
from PIL import Image as PILImage, ImageDraw, ImageFont

mcp = FastMCP("image-repro")

@mcp.tool()
def show_secret_word() -> Image:
"""Renders an image containing a secret word. Look at the image
and report the word."""
word = "".join(random.choices(string.ascii_uppercase, k=6))
img = PILImage.new("RGB", (600, 200), "white")
ImageDraw.Draw(img).text((40, 60), word, fill="black",
font=ImageFont.load_default(size=80))
buf = io.BytesIO(); img.save(buf, format="PNG")
return Image(data=buf.getvalue(), format="png")

mcp.run(transport="stdio")
```

1. Add to Claude Desktop, new chat.
2. Ask: **"Call show_secret_word and tell me the word in the image."**
3. Observed: the tool runs, the image with the word renders in the
collapsed tool panel — and the model answers with a **different,
invented word** (or an invented scene description for photographic
images).
4. Reply anything ("are you sure?").
5. Observed: the model now names the word **correctly** and usually
notes its previous answer was wrong.

Step 3 vs step 5 is deterministic across our sessions; with a
random word per call, priors cannot explain the correct second
answer — the image demonstrably reaches the model, one user turn
late.

## Ruled out (server-side hardening that changed nothing)

We shipped three successive result shapes to exclude prompt/attention
causes on our side:

1. `[info-JSON, image]` — model answered from the JSON text.
2. `[info-JSON with explicit "describe only the attached image"
anchor, image]` — still confabulated on the calling turn.
3. **Image alone**, all metadata burned into a caption bar inside the
pixels, zero text blocks — still confabulated on the calling turn
("macro photo of dew drops on a spider web" for a gray noise test
card reading "КЕДР-77" with a blue circle), then described the
card exactly on the next turn.

With (3) there is literally no text to answer from, so the calling
turn's context cannot contain the image. Result size is not the
cause either: all images are ≤ 700 KB encoded (we budget them), well
under the 1 MB tool-result cap.

## Expected

`ImageContent` from an MCP tool result should be part of the model's
context in the same turn that called the tool — as it already is in
Claude Code with the identical server.

## Impact

Any MCP server that returns images for the model to read (charts,
screenshots, rendered pages, OCR aids) silently produces confident
hallucinations in Desktop chat: the model describes what it expects
instead of what the tool returned, and the user has no way to tell.
An honest failure (no image in context → "I can't see it") would
already be an improvement; the actual behavior is worse than the
tool not existing.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。