cloudflare / cloudflare/cloudflare-os
MCP image content blocks can never reach the model: executeCode results are text-only
- Dominant language
- TypeScript
- Stars
- 9.9k
- Forks
- 1.2k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 107
Description
## Summary
MCP tool results can carry `{type: "image"}` content blocks, and the MCP gatekeepers faithfully deliver them to the agent's bindings — but the agent can never actually *see* them. The only route from a binding back into the model context is the `executeCode` tool, whose result is text-only, so every image block dies at that boundary. Any MCP server that answers with images (screenshots, charts, rendered pages, camera stills, PDF page renders) is invisible to the agent by construction.
This looks like a dangling end between two halves of the codebase that each already support images.
## The path, as it is today
1. `packages/mcp-shared/src/tools.ts` — `toCallResult()` preserves the full content-block array of a `tools/call` result, including `image` blocks. So `env.MY_MCP.someTool()` inside `executeCode` really does receive the image (base64 + mimeType).
2. `packages/workshop-backend/src/overseer.ts` — `executeCodeMode()` builds the tool result **from the collected console log alone** (the trace items joined into a string). The executed module's return value is discarded by `CODE_MODE_HARNESS` (`await agent(self, env, this.ctx);` — nothing is captured).
3. `packages/workshop-backend/src/agent.ts` — the `executeCode` tool wraps that string as `content: [{type: "text", ...}]`. Every other tool does the same.
So the best an agent can do is `console.log` the base64, which (a) is not vision — the model sees characters, not pixels; (b) is token-expensive; and (c) rides the whole result through storage as text.
## The pieces to fix it already exist
- The model layer is already multimodal **including tool results**: pi-ai's `ToolResultMessage.content` is `(TextContent | ImageContent)[]`, and the providers translate image parts.
- Chat attachments already reach the model as `ImageContent` (`agent.ts`).
- `agent-compaction.ts` already knows how to prune images from projections (replacing them with `[image ]` markers), so bounding image bytes in long contexts has a natural hook.
The gap is only the plumbing between `executeCode` and the `ToolResultMessage`.
## Proposed mechanism (sketch)
1. `CODE_MODE_HARNESS` captures the default function's **return value** and hands it back from `run()` (today it is discarded).
2. `executeCodeMode()` returns `{log, images}`: images extracted from the returned value when it contains MCP-style `{type: "image", data, mimeType}` blocks (or a small documented shape), with caps on count and total bytes.
3. `agent.ts` emits them as `ImageContent` parts in the tool result, after the log text.
4. Persistence: store only a short marker (e.g. `[image image/jpeg, 48 KB]`) in the stored tool output. Replayed history then shows the marker, mirroring what compaction already does to old images — Durable Object rows stay small, and images naturally live only within the run that produced them, which is exactly the window where an agent acts on what it saw.
With that in place, the docs could tell agents: "to look at an image a binding returned, `return` its content blocks from `executeCode`".
## Why it matters
Image-producing MCP servers are common (browser/screenshot servers, chart renderers, vision pipelines), and connecting them is this platform's headline feature — today they all silently degrade to text. This also unblocks screenshot-driven flows (a model observing a GUI through an MCP bridge) without any special-casing: it is the same generic mechanism.
Happy to contribute a PR if the direction sounds right.
Contributor guide
Research direction
Start with CODE_MODE_HARNESS and executeCodeMode() in packages/workshop-backend/src/overseer.ts, then follow the executeCode tool result construction in packages/workshop-backend/src/agent.ts and the content handling in packages/mcp-shared/src/tools.ts. Verify that returned MCP image blocks reach the model as ImageContent, while persisted or replayed output uses short markers and text-only results remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100