MCP tool result image content blocks are not surfaced to the model
まだ誰も着手していません。
- 主要言語
- Shell
- スター
- 11.2k
- フォーク
- 1.9k
- 平均マージ
- 14時間 16分
- マージ済み PR(30日)
- 6
説明
Describe the bug
When an MCP server returns an image content block in a tool result (per the MCP specification: content array item {type:"image", mimeType:"image/png", data:}), the image never reaches the model. Only the text blocks and structuredContent of the same tool result are delivered.
Verified from the agent side (Copilot CLI 1.0.80, Windows):
- The MCP server's wire format is spec-compliant. Capturing the raw stdio JSON-RPC response for a screenshot tool shows content[0] = text block and content[1] = a valid image block (mimeType image/png, correct base64 PNG payload).
-
- Across multiple calls (window screenshot, region screenshot), the image data never appeared in the model context; only the text metadata block arrived.
-
- Static inspection of app.js (1.0.80) shows image handling only in two paths: user-attachment sanitization (file_data / image_url / input_image) and the mcp-sampling converter for user-role messages. No conversion path exists for MCP tool-result image blocks, and tool results are assembled through the native session layer (session.mcp.apps.callTool).
User-attached images (paste / drag-drop / @file) work fine, so the model and plan support vision. The gap is specific to images returned by MCP tools.
- Static inspection of app.js (1.0.80) shows image handling only in two paths: user-attachment sanitization (file_data / image_url / input_image) and the mcp-sampling converter for user-role messages. No conversion path exists for MCP tool-result image blocks, and tool results are assembled through the native session layer (session.mcp.apps.callTool).
Affected version
1.0.80 (Windows 10.0.26200, PowerShell)
Steps to reproduce the behavior
- Configure any MCP server whose tool returns an image content block (e.g. a UI-automation server with a screenshot tool; any tool returning {type:"image", mimeType:"image/png", data:} reproduces it).
-
- In an interactive Copilot CLI session, ask the agent to call that tool and then describe what it sees in the image.
-
- Observe: the agent only receives the text blocks / structuredContent of the tool result. It reports the image metadata but cannot see the actual pixels, and often states the image content is unavailable.
Expected behavior
When the selected model supports vision, image content blocks inside MCP tool results should be passed to the model (like user-attached images), so agents can act on screenshots and other tool-generated images.
Additional context
This blocks a whole class of computer-use / UI-automation workflows: MCP servers that follow the spec and return screenshots as image blocks work in other agent CLIs that surface tool-result images, but not in Copilot CLI. Current workaround: agents must rely on structured reads only, or the user must manually attach the image.
Issue submitted by GLM-5.3 via rpacu MCP
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
調査の方向性
まず、既存のユーザー添付ファイルとMCP-samplingの画像処理から、session.mcp.apps.callToolを通じて組み立てられるツール結果まで、app.jsを追跡します。仕様に準拠した画像ブロックを返すMCPスクリーンショットツールで再現し、その後、視覚対応モデルが画像を受け取り、テキストメタデータだけでなく画像を説明できることを確認します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- javascript, shell
- 領域
- ai, cli
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 62/100