ag-ui-protocol / ag-ui-protocol/ag-ui

[Feature]: Add a provider file-handle arm to InputContentSource

Offen
#2,639 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
15.9k
Forks
1.4k
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
163

Beschreibung

### Problem

`InputContentSource` can express media as inline base64 (`data`) or as a URL (`url`), but not as a **provider-issued file handle**.

Every major provider now has a Files/storage API where you upload an asset once and reference it by an opaque handle on subsequent requests:

| Provider | Reference |
| --- | --- |
| OpenAI (Responses) | `input_image` / `input_file` with `file_id` |
| Anthropic | `{ type: "file", file_id }` message source (`files-api-2025-04-14`) |
| Gemini | `fileData.fileUri` |
| fal | storage URL |

These are not interchangeable with the existing two arms:

- An OpenAI or Anthropic `file_id` is an **opaque id**, not a URL, so it cannot go in `url`.
- A Gemini file URI *looks* like a URL but is auth-gated and only meaningful to Gemini — putting it in `url` invites any other consumer to try to fetch it.
- Putting either in `data` is simply wrong; there are no bytes.

The whole point of the handle is that the bytes are **not** re-sent, so `data` defeats the feature and `url` mis-describes it.

### Where this bites

In [TanStack AI](https://github.com/TanStack/ai) we added a `{ type: 'file' }` content source that carries a per-provider reference record, and adapters map it to each provider's native field. It works fine server-side, but our AG-UI serializer (`uiMessagesToWire` → `RunAgentInput.messages`) is the client → server transport for our `useChat` hook, and a file source cannot cross it. We currently throw a clear error there rather than silently dropping the part or mis-encoding it as a URL. That means the browser can't hold a handle and reference it across turns — the handle has to travel out-of-band in the caller's own payload and be rebuilt server-side, which routes around the protocol.

Any framework wiring a provider Files API through AG-UI will hit the same wall.

### Proposal

Add a third arm to `InputContentSourceSchema` in `sdks/typescript/packages/core/src/types.ts` (and the peer SDKs), alongside `InputContentDataSourceSchema` / `InputContentUrlSourceSchema`. Roughly:

```ts
const InputContentFileSourceSchema = z.object({
type: z.literal("file"),
// Provider name -> that provider's reference (file_id, file URI, storage URL).
reference: z.record(z.string(), z.string()),
mimeType: z.string().optional(),
})
```

Keying by provider matters: the same bytes uploaded to two providers merge into one source that routes correctly to either, and a consumer reads only its own entry and errors when there is none — rather than guessing whether an opaque string belongs to it. A single `{ provider, reference }` pair would also work if a record is more than the spec wants; the essential part is that the reference is **explicitly a provider handle**, distinguishable from a fetchable URL.

Since `InputContentSource` is a discriminated union validated at runtime, this needs a schema change, not just a type change — hence raising it here rather than working around it downstream.

### Related

- #126 — the original multimodal-transport proposal (closed); this is the natural follow-on now that providers reference media by handle rather than by value.
- #2133 — tracking spec-compliant `InputContent` across SDKs; a new arm would need to land there too.

### Versions

Checked `@ag-ui/core@0.0.59` (latest stable) and `0.1.1-canary.beta.0` — both are `data | url`.

### Happy to help

If the shape is agreeable I'm glad to open a PR for the TypeScript SDK (schema, types, tests) and follow up on the docs.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.