ag-ui-protocol / ag-ui-protocol/ag-ui

[Feature]: Symmetric multimodal output with typed `OutputContent[]` on `AssistantMessage`

Đang mở
#2,029 0 bình luận 5 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Python
Star
15.9k
Fork
1.4k
Merge trung bình
1 ngày 17 giờ
Pull request đã merge (30 ngày)
163

Mô tả

### Pre-flight Checklist

- [x] I have searched existing issues and this hasn't been requested yet.

### Problem or Motivation

AG-UI has formalized multimodal **input**: `UserMessage.content` accepts `string | InputContent[]`, with typed `Text` / `Image` / `Audio` / `Video` / `Document` parts and `data` / `url` sources.

The **output** side never followed. `AssistantMessage.content` is still `string` only. There is no typed way for an agent to send media (an image, an audio clip, a generated file) back to the user. The asymmetry is real and currently undefended: it was raised for *both* user and assistant messages in #126, but only the input half shipped, and the "what about messages *from* the agent?" question was asked twice in that thread and never answered before it was closed. The same need keeps resurfacing in discussions (#527 "Support for Images & Files", #168 "How to send ImageMessage", #526 "artifacts in A2A").

Today, agent-produced media is only possible through app-specific conventions: tool calls + generative UI, shared state, or `CUSTOM`/`RAW` events. Those are fine for a React frontend, but they leave 4 gaps that a *protocol* is supposed to close:

1. **Fragmentation.** Without a typed shape, every framework and app invents its own output-media convention. This is the exact concern that motivated #126 in the first place and it now applies to the half of the conversation that was left out.
2. **Non-UI consumers.** A2A agent-to-agent flows, CLIs, mobile clients, and eval/logging pipelines have no frontend to render a tool call into an `` or audio player. They need the media typed *in the message* to consume it at all.
3. **Native multimodal-output models.** Models that emit images/audio as first-class output (e.g. Gemini) have no clean channel. You must wrap genuine model output in a synthetic tool call.
4. **Round-trip fidelity.** Media that isn't part of the message model can't be faithfully represented in `MESSAGES_SNAPSHOT`, persisted, or replayed. (Media metadata is already lossy even on the *input* side; see #1809 and #2011 which suggests the message model under-specifies media generally.)

The result is that AG-UI, a protocol whose whole purpose is connecting agents to users, can carry rich media *to* the agent but not *from* it.

### Proposed Solution

Make output symmetric with input by allowing typed content parts on assistant messages:

```ts
interface AssistantMessage {
id: string
role: "assistant"
content?: string | OutputContent[] // was: string
toolCalls?: ToolCall[]
name?: string
encryptedContent?: string
}

type OutputContent =
| TextOutputContent
| ImageOutputContent
| AudioOutputContent
| VideoOutputContent
| DocumentOutputContent
```

Each non-text part carries a `source` and optional `metadata`, **reusing the existing input source types** rather than inventing new ones (ideally promoted to a single shared `ContentSource`):

```ts
// shared with the input side
type ContentSource =
| { type: "data"; value: string; mimeType: string } // inline / base64
| { type: "url"; value: string; mimeType?: string } // reference
```

Backward compatibility is preserved exactly as it was on the user side: `content` stays `string | OutputContent[]`, so existing text-only agents and clients are unaffected.

**Open design question: how it streams.** This is the main thing worth deciding together. `MESSAGES_SNAPSHOT` can already carry the assembled message, but live streaming today is text-only (`TEXT_MESSAGE_*`). A pragmatic v1 could deliver each media part as a **discrete, complete part** (URL-first to avoid large base64 over the stream) via a single new content-part event, deferring true inline/streamed binary to a later iteration. I'd rather align on the approach than prescribe one.

Happy to hear from you, learn and adjust.

### Alternatives Considered

- **Tool calls + generative UI (status quo).** Great for rich React frontends and the right tool for *interactive* output. But it's an app-level convention, not portable across frameworks, and invisible to any non-UI consumer.
- **Shared state (`STATE_SNAPSHOT` / `STATE_DELTA`).** Good for persistent artifact panels, but it's a side channel, the media isn't part of the turn, so association and ordering with the message are implicit.
- **`CUSTOM` / `RAW` events.** The escape hatch that works today and precisely the fragmentation #126 warned about.
- **A2A artifacts (#526).** Solves a different layer; doesn't address the AG-UI message model itself.
- **Markdown / `data:` URIs inside the `content` string.** Fragile, untyped, no metadata, and unworkable for audio/video.

These each have a legitimate place; none of them gives the protocol a standard, interoperable representation of agent-produced media, which is what this proposal is about.

### Additional Context

_No response_

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.