ag-ui-protocol / ag-ui-protocol/ag-ui

RFC: A2UI bulk data by reference (avoid paying output-token rates to render server-held data)

Đang mở
#2,005 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement proposal
Ngôn ngữ chính
Python
Star
15.9k
Fork
1.4k
Merge trung bình
1 ngày 17 giờ
Pull request đã merge (30 ngày)
163

Mô tả

## TL;DR

To render a data surface (table, chart, roster), A2UI currently requires the **model** to emit the entire dataset inline as `render_a2ui` tool-call arguments. Path binding only references the model's *own* output data model — there's no way to bind a surface to data the **agent already holds**. So a 1,000-row table the agent fetched in milliseconds gets re-serialized as **~27K output tokens** before the surface can render: large latency, output-token cost, and a redundant data hop.

This proposes an **additive, opt-in** path: the agent supplies bulk data out-of-band; the middleware injects it into the surface's data model via `updateDataModel`. The model emits only the component structure + a path reference — never the rows. Inline behavior is unchanged, so cross-SDK parity holds.

## Problem

`render_a2ui` takes `components` + a `dataModel`. A `List`/`Row` binds children to a data-model path (e.g. `/rows`), but the rows live in `dataModel.rows`, **inlined in the model's tool-call args** (the middleware extracts them from streaming `TOOL_CALL_ARGS`). Bindings resolve only against that model-produced data model — nothing points at server-held data. So every displayed row must be generated by the LLM as output tokens, even when the agent already has the data.

### Measured cost (deterministic emitter, no model variance)

Emitting the exact `render_a2ui` call for an N-row table:

| Rows | `render_a2ui` args | ≈ output tokens |
|-----:|-------------------:|----------------:|
| 10 | 1,350 B | ~338 |
| 200 | 22,067 B | ~5,517 |
| 1,000| 109,630 B | ~27,408 |

Output-token cost scales linearly with the dataset. At real generation rates this is the "tens of seconds to render a table I already have" that users hit. The data also makes a wasteful middle hop (agent → model output → surface) paid at output-token rates, with transcription-drift risk on large datasets.

**Related transport cost (compounds this):** `render_a2ui` data-model updates are emitted as **cumulative per-row snapshots** during streaming, so a K-row surface ships ~`1+2+…+K = O(K²)` wire bytes (megabytes for a large repeated-data surface). By-reference data delivered as a single `updateDataModel` sidesteps both the output-token cost *and* this O(K²) snapshot amplification.

## Proposed design (additive, opt-in)

The middleware already understands `updateDataModel` (`{ surfaceId, path, value }`) — today only the model produces it. Let the **agent** produce it:

1. **Agent declares data out-of-band** — attach `{ surfaceId, path, value }` via `forwardedProps.a2uiData` (mirrors the existing `forwardedProps.a2uiAction` the middleware already reads) and/or agent state.
2. **Model emits structure only** — `render_a2ui` with components (a `List` bound to `/rows`), omitting the rows from `dataModel`. Tool guidelines: "when data is provided by reference, bind to its path; don't inline it."
3. **Middleware merges** — emits the model's `updateComponents` plus an `updateDataModel` built from the agent payload. The renderer already resolves path bindings against the data model (same mechanism as form pre-fill) — **no renderer change**.

Result: `render_a2ui` args carry structure + a path reference (tens of tokens), not the dataset.

### Why it's safe
- **Additive**: no `a2uiData` ⇒ identical to today (model inlines, middleware extracts). No existing test/integration changes.
- **Parity-preserving**: `updateDataModel` is already in the cross-SDK schema (TS/Python/.NET); we add a *producer*, not a new wire contract.
- **No new render semantics**: path-binding against the data model already works.

## Scope
- **spec**: document the by-reference contract (no breaking change to the op shape).
- **`@ag-ui/a2ui-middleware`**: read agent-supplied data; emit merged `updateDataModel`; update `RENDER_A2UI_TOOL_GUIDELINES`.
- **adapters**: typed helper to attach `a2uiData` (Strands / LangGraph first).
- **dojo + e2e**: an `a2uiByReference` feature asserting structure-only args render a populated surface and the dataset is absent from model output.

## Open questions
1. **Channel**: `forwardedProps.a2uiData` (least invasive, side-channel) vs. a first-class schema field (cleaner, bigger change)?
2. **Large datasets**: by-reference removes output-token cost but the payload still crosses the wire once — streaming/windowing is follow-up.
3. **Validation**: happy to build a middleware + Strands spike that re-measures against the table above (expect ~flat args as N grows) once there's buy-in on the approach + channel.

Sourced from real user reports of data-heavy A2UI surfaces. Looking for maintainer feedback on the approach and the channel choice before a spike.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.