ag-ui-protocol / ag-ui-protocol/ag-ui

[Feature]: Symmetric multimodal output with typed `OutputContent[]` on `AssistantMessage`

未關閉
#2,029 0 則留言 5 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
15.9k
分支
1.4k
平均合併
1 天 17 小時
30 天內合併 PR
163

描述

### Pre-flight Checklist

- [x] I have searched existing issues and this hasn't been requested yet.

### Problem or Motivation

AG-UI has formalized multimodal **input**: `UserMessage.content` accepts `string | InputContent[]`, with typed `Text` / `Image` / `Audio` / `Video` / `Document` parts and `data` / `url` sources.

The **output** side never followed. `AssistantMessage.content` is still `string` only. There is no typed way for an agent to send media (an image, an audio clip, a generated file) back to the user. The asymmetry is real and currently undefended: it was raised for *both* user and assistant messages in #126, but only the input half shipped, and the "what about messages *from* the agent?" question was asked twice in that thread and never answered before it was closed. The same need keeps resurfacing in discussions (#527 "Support for Images & Files", #168 "How to send ImageMessage", #526 "artifacts in A2A").

Today, agent-produced media is only possible through app-specific conventions: tool calls + generative UI, shared state, or `CUSTOM`/`RAW` events. Those are fine for a React frontend, but they leave 4 gaps that a *protocol* is supposed to close:

1. **Fragmentation.** Without a typed shape, every framework and app invents its own output-media convention. This is the exact concern that motivated #126 in the first place and it now applies to the half of the conversation that was left out.
2. **Non-UI consumers.** A2A agent-to-agent flows, CLIs, mobile clients, and eval/logging pipelines have no frontend to render a tool call into an `` or audio player. They need the media typed *in the message* to consume it at all.
3. **Native multimodal-output models.** Models that emit images/audio as first-class output (e.g. Gemini) have no clean channel. You must wrap genuine model output in a synthetic tool call.
4. **Round-trip fidelity.** Media that isn't part of the message model can't be faithfully represented in `MESSAGES_SNAPSHOT`, persisted, or replayed. (Media metadata is already lossy even on the *input* side; see #1809 and #2011 which suggests the message model under-specifies media generally.)

The result is that AG-UI, a protocol whose whole purpose is connecting agents to users, can carry rich media *to* the agent but not *from* it.

### Proposed Solution

Make output symmetric with input by allowing typed content parts on assistant messages:

```ts
interface AssistantMessage {
id: string
role: "assistant"
content?: string | OutputContent[] // was: string
toolCalls?: ToolCall[]
name?: string
encryptedContent?: string
}

type OutputContent =
| TextOutputContent
| ImageOutputContent
| AudioOutputContent
| VideoOutputContent
| DocumentOutputContent
```

Each non-text part carries a `source` and optional `metadata`, **reusing the existing input source types** rather than inventing new ones (ideally promoted to a single shared `ContentSource`):

```ts
// shared with the input side
type ContentSource =
| { type: "data"; value: string; mimeType: string } // inline / base64
| { type: "url"; value: string; mimeType?: string } // reference
```

Backward compatibility is preserved exactly as it was on the user side: `content` stays `string | OutputContent[]`, so existing text-only agents and clients are unaffected.

**Open design question: how it streams.** This is the main thing worth deciding together. `MESSAGES_SNAPSHOT` can already carry the assembled message, but live streaming today is text-only (`TEXT_MESSAGE_*`). A pragmatic v1 could deliver each media part as a **discrete, complete part** (URL-first to avoid large base64 over the stream) via a single new content-part event, deferring true inline/streamed binary to a later iteration. I'd rather align on the approach than prescribe one.

Happy to hear from you, learn and adjust.

### Alternatives Considered

- **Tool calls + generative UI (status quo).** Great for rich React frontends and the right tool for *interactive* output. But it's an app-level convention, not portable across frameworks, and invisible to any non-UI consumer.
- **Shared state (`STATE_SNAPSHOT` / `STATE_DELTA`).** Good for persistent artifact panels, but it's a side channel, the media isn't part of the turn, so association and ordering with the message are implicit.
- **`CUSTOM` / `RAW` events.** The escape hatch that works today and precisely the fragmentation #126 warned about.
- **A2A artifacts (#526).** Solves a different layer; doesn't address the AG-UI message model itself.
- **Markdown / `data:` URIs inside the `content` string.** Fragile, untyped, no metadata, and unworkable for audio/video.

These each have a legitimate place; none of them gives the protocol a standard, interoperable representation of agent-produced media, which is what this proposal is about.

### Additional Context

_No response_

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。