ag-ui-protocol / ag-ui-protocol/ag-ui

[Feature]: Add a provider file-handle arm to InputContentSource

Aperta
#2,639 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
15.9k
Fork
1.4k
Merge medio
1g 17h
PR unite (30g)
163

Descrizione

### Problem

`InputContentSource` can express media as inline base64 (`data`) or as a URL (`url`), but not as a **provider-issued file handle**.

Every major provider now has a Files/storage API where you upload an asset once and reference it by an opaque handle on subsequent requests:

| Provider | Reference |
| --- | --- |
| OpenAI (Responses) | `input_image` / `input_file` with `file_id` |
| Anthropic | `{ type: "file", file_id }` message source (`files-api-2025-04-14`) |
| Gemini | `fileData.fileUri` |
| fal | storage URL |

These are not interchangeable with the existing two arms:

- An OpenAI or Anthropic `file_id` is an **opaque id**, not a URL, so it cannot go in `url`.
- A Gemini file URI *looks* like a URL but is auth-gated and only meaningful to Gemini — putting it in `url` invites any other consumer to try to fetch it.
- Putting either in `data` is simply wrong; there are no bytes.

The whole point of the handle is that the bytes are **not** re-sent, so `data` defeats the feature and `url` mis-describes it.

### Where this bites

In [TanStack AI](https://github.com/TanStack/ai) we added a `{ type: 'file' }` content source that carries a per-provider reference record, and adapters map it to each provider's native field. It works fine server-side, but our AG-UI serializer (`uiMessagesToWire` → `RunAgentInput.messages`) is the client → server transport for our `useChat` hook, and a file source cannot cross it. We currently throw a clear error there rather than silently dropping the part or mis-encoding it as a URL. That means the browser can't hold a handle and reference it across turns — the handle has to travel out-of-band in the caller's own payload and be rebuilt server-side, which routes around the protocol.

Any framework wiring a provider Files API through AG-UI will hit the same wall.

### Proposal

Add a third arm to `InputContentSourceSchema` in `sdks/typescript/packages/core/src/types.ts` (and the peer SDKs), alongside `InputContentDataSourceSchema` / `InputContentUrlSourceSchema`. Roughly:

```ts
const InputContentFileSourceSchema = z.object({
type: z.literal("file"),
// Provider name -> that provider's reference (file_id, file URI, storage URL).
reference: z.record(z.string(), z.string()),
mimeType: z.string().optional(),
})
```

Keying by provider matters: the same bytes uploaded to two providers merge into one source that routes correctly to either, and a consumer reads only its own entry and errors when there is none — rather than guessing whether an opaque string belongs to it. A single `{ provider, reference }` pair would also work if a record is more than the spec wants; the essential part is that the reference is **explicitly a provider handle**, distinguishable from a fetchable URL.

Since `InputContentSource` is a discriminated union validated at runtime, this needs a schema change, not just a type change — hence raising it here rather than working around it downstream.

### Related

- #126 — the original multimodal-transport proposal (closed); this is the natural follow-on now that providers reference media by handle rather than by value.
- #2133 — tracking spec-compliant `InputContent` across SDKs; a new arm would need to land there too.

### Versions

Checked `@ag-ui/core@0.0.59` (latest stable) and `0.1.1-canary.beta.0` — both are `data | url`.

### Happy to help

If the shape is agreeable I'm glad to open a PR for the TypeScript SDK (schema, types, tests) and follow up on the docs.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.