anthropics / anthropics/claude-ai-mcp

MCP tool-result images are invisible to the model during the calling turn (Claude Desktop) — model confabulates, then describes correctly on the next user message

オープン
#643 コメント 5 件 リアクション 4 件 担当者 0 名 GitHub で見る
主要言語
言語のデータがありません
スター
471
フォーク
77
PR マージ指標
30日以内にマージされた PR はありません

説明

## Summary

In Claude Desktop chat, when an MCP tool returns `ImageContent`, the
image renders in the tool-call panel in the UI, but the model's
continuation of the same agentic turn behaves as if the image block is
absent: it reliably "describes" the image from priors (file name,
conversation context, memory). On the **next user message** the model
suddenly sees the image and describes it accurately, often
apologizing for the previous answer. The same server's images are
visible to the model within the calling turn in Claude Code, so this
looks specific to the Desktop/claude.ai chat context assembly.

## Environment

- Claude Desktop 1.22209.0, macOS 15.0.1 (Apple Silicon)
- Chat model: Claude Opus 4.8 (also reproduced with default model)
- MCP server: perception-mcp 0.1.4 → 0.1.7 (PyPI), Python MCP SDK
(FastMCP) 1.28.1, stdio transport, launched via uvx
- Both `image/png` and `image/jpeg` content blocks affected equally

## Repro

Minimal server (20 lines, no dependencies beyond `mcp` + `pillow`):

```python
# repro_server.py
import io, random, string
from mcp.server.fastmcp import FastMCP, Image
from PIL import Image as PILImage, ImageDraw, ImageFont

mcp = FastMCP("image-repro")

@mcp.tool()
def show_secret_word() -> Image:
"""Renders an image containing a secret word. Look at the image
and report the word."""
word = "".join(random.choices(string.ascii_uppercase, k=6))
img = PILImage.new("RGB", (600, 200), "white")
ImageDraw.Draw(img).text((40, 60), word, fill="black",
font=ImageFont.load_default(size=80))
buf = io.BytesIO(); img.save(buf, format="PNG")
return Image(data=buf.getvalue(), format="png")

mcp.run(transport="stdio")
```

1. Add to Claude Desktop, new chat.
2. Ask: **"Call show_secret_word and tell me the word in the image."**
3. Observed: the tool runs, the image with the word renders in the
collapsed tool panel — and the model answers with a **different,
invented word** (or an invented scene description for photographic
images).
4. Reply anything ("are you sure?").
5. Observed: the model now names the word **correctly** and usually
notes its previous answer was wrong.

Step 3 vs step 5 is deterministic across our sessions; with a
random word per call, priors cannot explain the correct second
answer — the image demonstrably reaches the model, one user turn
late.

## Ruled out (server-side hardening that changed nothing)

We shipped three successive result shapes to exclude prompt/attention
causes on our side:

1. `[info-JSON, image]` — model answered from the JSON text.
2. `[info-JSON with explicit "describe only the attached image"
anchor, image]` — still confabulated on the calling turn.
3. **Image alone**, all metadata burned into a caption bar inside the
pixels, zero text blocks — still confabulated on the calling turn
("macro photo of dew drops on a spider web" for a gray noise test
card reading "КЕДР-77" with a blue circle), then described the
card exactly on the next turn.

With (3) there is literally no text to answer from, so the calling
turn's context cannot contain the image. Result size is not the
cause either: all images are ≤ 700 KB encoded (we budget them), well
under the 1 MB tool-result cap.

## Expected

`ImageContent` from an MCP tool result should be part of the model's
context in the same turn that called the tool — as it already is in
Claude Code with the identical server.

## Impact

Any MCP server that returns images for the model to read (charts,
screenshots, rendered pages, OCR aids) silently produces confident
hallucinations in Desktop chat: the model describes what it expects
instead of what the tool returned, and the user has no way to tell.
An honest failure (no image in context → "I can't see it") would
already be an improvement; the actual behavior is worse than the
tool not existing.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with the minimal repro in repro_server.py and run it through Claude Desktop using the stated steps. Compare the model's response in the calling turn with its response after a follow-up message, and check the same server in Claude Code. Done means ImageContent is available to the model during the original tool-calling turn and the image is described accurately.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
ai, api
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
おおむね明確
初心者へのやさしさ
38/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。