ag-ui-protocol / ag-ui-protocol/ag-ui
RFC: A2UI bulk data by reference (avoid paying output-token rates to render server-held data)
- Ngôn ngữ chính
- Python
- Star
- 15.9k
- Fork
- 1.4k
- Merge trung bình
- 1 ngày 17 giờ
- Pull request đã merge (30 ngày)
- 163
Mô tả
## TL;DR
To render a data surface (table, chart, roster), A2UI currently requires the **model** to emit the entire dataset inline as `render_a2ui` tool-call arguments. Path binding only references the model's *own* output data model — there's no way to bind a surface to data the **agent already holds**. So a 1,000-row table the agent fetched in milliseconds gets re-serialized as **~27K output tokens** before the surface can render: large latency, output-token cost, and a redundant data hop.
This proposes an **additive, opt-in** path: the agent supplies bulk data out-of-band; the middleware injects it into the surface's data model via `updateDataModel`. The model emits only the component structure + a path reference — never the rows. Inline behavior is unchanged, so cross-SDK parity holds.
## Problem
`render_a2ui` takes `components` + a `dataModel`. A `List`/`Row` binds children to a data-model path (e.g. `/rows`), but the rows live in `dataModel.rows`, **inlined in the model's tool-call args** (the middleware extracts them from streaming `TOOL_CALL_ARGS`). Bindings resolve only against that model-produced data model — nothing points at server-held data. So every displayed row must be generated by the LLM as output tokens, even when the agent already has the data.
### Measured cost (deterministic emitter, no model variance)
Emitting the exact `render_a2ui` call for an N-row table:
| Rows | `render_a2ui` args | ≈ output tokens |
|-----:|-------------------:|----------------:|
| 10 | 1,350 B | ~338 |
| 200 | 22,067 B | ~5,517 |
| 1,000| 109,630 B | ~27,408 |
Output-token cost scales linearly with the dataset. At real generation rates this is the "tens of seconds to render a table I already have" that users hit. The data also makes a wasteful middle hop (agent → model output → surface) paid at output-token rates, with transcription-drift risk on large datasets.
**Related transport cost (compounds this):** `render_a2ui` data-model updates are emitted as **cumulative per-row snapshots** during streaming, so a K-row surface ships ~`1+2+…+K = O(K²)` wire bytes (megabytes for a large repeated-data surface). By-reference data delivered as a single `updateDataModel` sidesteps both the output-token cost *and* this O(K²) snapshot amplification.
## Proposed design (additive, opt-in)
The middleware already understands `updateDataModel` (`{ surfaceId, path, value }`) — today only the model produces it. Let the **agent** produce it:
1. **Agent declares data out-of-band** — attach `{ surfaceId, path, value }` via `forwardedProps.a2uiData` (mirrors the existing `forwardedProps.a2uiAction` the middleware already reads) and/or agent state.
2. **Model emits structure only** — `render_a2ui` with components (a `List` bound to `/rows`), omitting the rows from `dataModel`. Tool guidelines: "when data is provided by reference, bind to its path; don't inline it."
3. **Middleware merges** — emits the model's `updateComponents` plus an `updateDataModel` built from the agent payload. The renderer already resolves path bindings against the data model (same mechanism as form pre-fill) — **no renderer change**.
Result: `render_a2ui` args carry structure + a path reference (tens of tokens), not the dataset.
### Why it's safe
- **Additive**: no `a2uiData` ⇒ identical to today (model inlines, middleware extracts). No existing test/integration changes.
- **Parity-preserving**: `updateDataModel` is already in the cross-SDK schema (TS/Python/.NET); we add a *producer*, not a new wire contract.
- **No new render semantics**: path-binding against the data model already works.
## Scope
- **spec**: document the by-reference contract (no breaking change to the op shape).
- **`@ag-ui/a2ui-middleware`**: read agent-supplied data; emit merged `updateDataModel`; update `RENDER_A2UI_TOOL_GUIDELINES`.
- **adapters**: typed helper to attach `a2uiData` (Strands / LangGraph first).
- **dojo + e2e**: an `a2uiByReference` feature asserting structure-only args render a populated surface and the dataset is absent from model output.
## Open questions
1. **Channel**: `forwardedProps.a2uiData` (least invasive, side-channel) vs. a first-class schema field (cleaner, bigger change)?
2. **Large datasets**: by-reference removes output-token cost but the payload still crosses the wire once — streaming/windowing is follow-up.
3. **Validation**: happy to build a middleware + Strands spike that re-measures against the table above (expect ~flat args as N grows) once there's buy-in on the approach + channel.
Sourced from real user reports of data-heavy A2UI surfaces. Looking for maintainer feedback on the approach and the channel choice before a spike.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.