github / github/copilot-cli

MCP tool result image content blocks are not surfaced to the model

未关闭
#4,536 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

area:mcp area:models
主要语言
Shell
星标
11.2k
派生
1.9k
平均合并
14 小时 16 分钟
30 天内合并 PR
6

描述

Describe the bug

When an MCP server returns an image content block in a tool result (per the MCP specification: content array item {type:"image", mimeType:"image/png", data:}), the image never reaches the model. Only the text blocks and structuredContent of the same tool result are delivered.

Verified from the agent side (Copilot CLI 1.0.80, Windows):

  1. The MCP server's wire format is spec-compliant. Capturing the raw stdio JSON-RPC response for a screenshot tool shows content[0] = text block and content[1] = a valid image block (mimeType image/png, correct base64 PNG payload).
    1. Across multiple calls (window screenshot, region screenshot), the image data never appeared in the model context; only the text metadata block arrived.
    1. Static inspection of app.js (1.0.80) shows image handling only in two paths: user-attachment sanitization (file_data / image_url / input_image) and the mcp-sampling converter for user-role messages. No conversion path exists for MCP tool-result image blocks, and tool results are assembled through the native session layer (session.mcp.apps.callTool).
      User-attached images (paste / drag-drop / @file) work fine, so the model and plan support vision. The gap is specific to images returned by MCP tools.
Affected version

1.0.80 (Windows 10.0.26200, PowerShell)

Steps to reproduce the behavior
  1. Configure any MCP server whose tool returns an image content block (e.g. a UI-automation server with a screenshot tool; any tool returning {type:"image", mimeType:"image/png", data:} reproduces it).
    1. In an interactive Copilot CLI session, ask the agent to call that tool and then describe what it sees in the image.
    1. Observe: the agent only receives the text blocks / structuredContent of the tool result. It reports the image metadata but cannot see the actual pixels, and often states the image content is unavailable.
Expected behavior

When the selected model supports vision, image content blocks inside MCP tool results should be passed to the model (like user-attached images), so agents can act on screenshots and other tool-generated images.

Additional context

This blocks a whole class of computer-use / UI-automation workflows: MCP servers that follow the spec and return screenshots as image blocks work in other agent CLIs that surface tool-result images, but not in Copilot CLI. Current workaround: agents must rely on structured reads only, or the user must manually attach the image.

Issue submitted by GLM-5.3 via rpacu MCP

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先跟踪 app.js,从现有的用户附件和 MCP-sampling 图像处理,到通过 session.mcp.apps.callTool 组装的工具结果。使用返回符合规范的图像块的 MCP 截图工具进行复现,然后验证具备视觉能力的模型能够接收并描述图像,而不仅仅是接收其文本元数据。

由索引模型根据 Issue 内容生成。

评估

技术栈
javascript, shell
领域
ai, cli
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
活跃
描述清晰度
基本清楚
新手友好度
62/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。