ag-ui-protocol / ag-ui-protocol/ag-ui
[Feature]: Symmetric multimodal output with typed `OutputContent[]` on `AssistantMessage`
- Langage dominant
- Python
- Étoiles
- 15.9k
- Forks
- 1.4k
- Merge moyen
- 1 j 17 h
- PR mergées (30 j)
- 163
Description
### Pre-flight Checklist
- [x] I have searched existing issues and this hasn't been requested yet.
### Problem or Motivation
AG-UI has formalized multimodal **input**: `UserMessage.content` accepts `string | InputContent[]`, with typed `Text` / `Image` / `Audio` / `Video` / `Document` parts and `data` / `url` sources.
The **output** side never followed. `AssistantMessage.content` is still `string` only. There is no typed way for an agent to send media (an image, an audio clip, a generated file) back to the user. The asymmetry is real and currently undefended: it was raised for *both* user and assistant messages in #126, but only the input half shipped, and the "what about messages *from* the agent?" question was asked twice in that thread and never answered before it was closed. The same need keeps resurfacing in discussions (#527 "Support for Images & Files", #168 "How to send ImageMessage", #526 "artifacts in A2A").
Today, agent-produced media is only possible through app-specific conventions: tool calls + generative UI, shared state, or `CUSTOM`/`RAW` events. Those are fine for a React frontend, but they leave 4 gaps that a *protocol* is supposed to close:
1. **Fragmentation.** Without a typed shape, every framework and app invents its own output-media convention. This is the exact concern that motivated #126 in the first place and it now applies to the half of the conversation that was left out.
2. **Non-UI consumers.** A2A agent-to-agent flows, CLIs, mobile clients, and eval/logging pipelines have no frontend to render a tool call into an `` or audio player. They need the media typed *in the message* to consume it at all.
3. **Native multimodal-output models.** Models that emit images/audio as first-class output (e.g. Gemini) have no clean channel. You must wrap genuine model output in a synthetic tool call.
4. **Round-trip fidelity.** Media that isn't part of the message model can't be faithfully represented in `MESSAGES_SNAPSHOT`, persisted, or replayed. (Media metadata is already lossy even on the *input* side; see #1809 and #2011 which suggests the message model under-specifies media generally.)
The result is that AG-UI, a protocol whose whole purpose is connecting agents to users, can carry rich media *to* the agent but not *from* it.
### Proposed Solution
Make output symmetric with input by allowing typed content parts on assistant messages:
```ts
interface AssistantMessage {
id: string
role: "assistant"
content?: string | OutputContent[] // was: string
toolCalls?: ToolCall[]
name?: string
encryptedContent?: string
}
type OutputContent =
| TextOutputContent
| ImageOutputContent
| AudioOutputContent
| VideoOutputContent
| DocumentOutputContent
```
Each non-text part carries a `source` and optional `metadata`, **reusing the existing input source types** rather than inventing new ones (ideally promoted to a single shared `ContentSource`):
```ts
// shared with the input side
type ContentSource =
| { type: "data"; value: string; mimeType: string } // inline / base64
| { type: "url"; value: string; mimeType?: string } // reference
```
Backward compatibility is preserved exactly as it was on the user side: `content` stays `string | OutputContent[]`, so existing text-only agents and clients are unaffected.
**Open design question: how it streams.** This is the main thing worth deciding together. `MESSAGES_SNAPSHOT` can already carry the assembled message, but live streaming today is text-only (`TEXT_MESSAGE_*`). A pragmatic v1 could deliver each media part as a **discrete, complete part** (URL-first to avoid large base64 over the stream) via a single new content-part event, deferring true inline/streamed binary to a later iteration. I'd rather align on the approach than prescribe one.
Happy to hear from you, learn and adjust.
### Alternatives Considered
- **Tool calls + generative UI (status quo).** Great for rich React frontends and the right tool for *interactive* output. But it's an app-level convention, not portable across frameworks, and invisible to any non-UI consumer.
- **Shared state (`STATE_SNAPSHOT` / `STATE_DELTA`).** Good for persistent artifact panels, but it's a side channel, the media isn't part of the turn, so association and ordering with the message are implicit.
- **`CUSTOM` / `RAW` events.** The escape hatch that works today and precisely the fragmentation #126 warned about.
- **A2A artifacts (#526).** Solves a different layer; doesn't address the AG-UI message model itself.
- **Markdown / `data:` URIs inside the `content` string.** Fragile, untyped, no metadata, and unworkable for audio/video.
These each have a legitimate place; none of them gives the protocol a standard, interoperable representation of agent-produced media, which is what this proposal is about.
### Additional Context
_No response_
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.