ag-ui-protocol / ag-ui-protocol/ag-ui
[Feature]: Multimodal tool results — widen `ToolMessage.content` to accept typed content parts
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.4k
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 163
Description
### Pre-flight Checklist
- [x] I have searched [existing issues](https://github.com/ag-ui-protocol/ag-ui/issues) and this hasn't been requested yet.
### Problem or Motivation
`ToolMessage.content` is typed `string`. A tool that produces an image, a document, or audio has no way to hand that payload back to the **model**, because the only field it can travel in is text.
This is a distinct gap from #2029. That issue is about agent-produced media reaching the **user** (`AssistantMessage`). This one is about tool-produced media reaching the **model** on the next inference step. Different direction, different consumer, and fixing either one does not fix the other.
The gap is already visible downstream in three places:
1. **Adapters serialize media into the string because there is nowhere else to put it.** #2233 was closed by making the aws-strands adapters stuff a non-text block's serialized payload into `content` rather than dropping it. That restores the bytes on the wire, but the payload is still text by the time a model sees it.
2. **#2632** reports `@ag-ui/claude-agent-sdk` truncating a multi-block tool result to `content[0]`. A tool result that is genuinely a text block plus an image block has no lossless target shape to be converted *into*, so every adapter picks its own lossy rule.
3. **Downstream consumers are ready and blocked.** In CopilotKit the runtime converts an AG-UI tool message to the Vercel AI SDK as `output: { type: "text", value: content }`, because `content` is a string. The AI SDK provider spec already accepts `output: { type: "content", value: [{ type: "file-data" | "file-url", … }] }`, and Anthropic's API accepts image blocks inside `tool_result`. The model-facing half exists; the protocol is the narrow part.
Reported downstream by a user at CopilotKit/CopilotKit#2264, who wanted an image-search tool to return images to a multimodal model and found no shape that works. The only workaround is to append a separate `UserMessage` carrying the media, which misattributes tool output as something the user said and breaks the tool-call/tool-result pairing.
### Proposed Solution
Widen `ToolMessage.content` the same way `UserMessage.content` was widened, reusing the existing source types rather than inventing new ones:
```ts
interface ToolMessage {
id: string
role: "tool"
content: string | ToolResultContent[] // was: string
toolCallId: string
error?: string
}
type ToolResultContent =
| TextInputContent
| ImageInputPart
| AudioInputPart
| VideoInputPart
| DocumentInputPart
```
Reusing `InputContent` parts is deliberate: a tool result and a user message are both content flowing *toward* the model, so they want the same `data` / `url` source shapes and the same metadata. If the alias reads oddly, `ToolResultContent` can be a distinct name over the same members.
`string` stays valid, so this is additive and every existing producer and consumer keeps working.
`TOOL_CALL_RESULT.content` needs the same widening to carry the parts on the streaming path, otherwise the shape only survives in `MESSAGES_SNAPSHOT`.
### Alternatives Considered
- **Serialize media into the `content` string** (what #2233 settled on). Keeps bytes on the wire, but the model receives text. It also leaves each adapter to invent its own envelope, which is the fragmentation a protocol is meant to prevent.
- **Emit a follow-up `UserMessage` with the media.** Works today and is what CopilotKit users are told to do, but it attributes tool output to the user and detaches the media from the tool call that produced it.
- **Tool calls plus generative UI.** Solves display, not model input. The model still never sees the image.
- **Leave it to `CUSTOM` / `RAW`.** Same objection as in #2029: the escape hatch is the fragmentation.
### Additional Context
Related: #2029 (the assistant-message half of multimodal output), #2233 (adapters dropping non-text tool result content), #2632 (multi-block tool results truncated), #2011 (media type and metadata lost on the input round-trip).
Contributor guide
Assessment
This issue has not been assessed yet.