feat(ai): native Files API support across providers (upload adapters + `file` content source)
- Dominant language
- TypeScript
- Stars
- 3.1k
- Forks
- 331
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 160
Description
Summary
Add first-class support for provider Files / storage APIs so callers can upload media once and reference it by a provider-issued handle, instead of re-sending base64 or relying on the provider to fetch a public URL every request.
Motivated by the discussion on #907 (Gemini image URL handling) and the original ask from @L4Ph about createGeminiImage() supporting the Files API.
Why
Today ContentPartSource is a two-arm union — { type: 'data' } (inline base64) or { type: 'url' } (HTTP(S)/gs:///data URI). That covers a lot, but it can't express a provider file handle, and it doesn't give us a place to hang native upload.
Benefits: large/reused inputs uploaded once (lower latency + bandwidth), no re-buffering of base64 on memory-constrained runtimes (the #907 OOM class), and access to provider-side file lifecycle (TTL, deletion).
Provider support (verified against the vendored SDKs in this repo)
| Provider | Native upload surface | Handle referenced as | Notes |
|---|---|---|---|
| Gemini | ai.files.upload() |
fileData.fileUri (a generativelanguage.googleapis.com/... URL) |
Handle is a URL, so it already round-trips through { type: 'url' } today (image-gen fixed in #907). |
| OpenAI | openai.files.upload() |
input_image / input_file file_id (file-...) |
ID is not a URL — no representation today. |
| Anthropic | client.beta.files.upload() (beta) |
file_id message source |
ID is not a URL — no representation today. |
| fal | fal.storage.upload() |
resulting storage URL | fal-client already auto-uploads Blob/File inputs (see ai-fal/src/utils/client.ts); we convert data: URLs to Blobs so it can. Could be surfaced explicitly. |
| xAI / Grok | (no dedicated files API found) | image.url / images[].url |
Grok's image edits accept URLs directly (passthrough), so no upload step is needed for URL inputs. |
Proposed shape
1. Tree-shakeable files adapter kind (following the existing Text/Embed/Image split), for providers that have one:
import { geminiFiles } from '@tanstack/ai-gemini/adapters'
const files = geminiFiles()
const handle = await files.upload({ data, mimeType }) // → { id, uri?, expiresAt? }
await files.get(handle.id)
await files.delete(handle.id)
openaiFiles(), anthropicFiles(), falFiles() (wrapping fal.storage.upload) likewise. Grok has nothing to wrap — URL inputs already work.
2. New { type: 'file' } arm on ContentPartSource:
{ type: 'image', source: { type: 'file', value: fileIdOrUri, provider: 'openai' } }
Per-provider typing should enforce that a file reference only flows to the provider that issued it (a file-... ID sent to Gemini is a bug). Adapters map it to the right wire field (file_id / fileData.fileUri).
3. Optional follow-up: an auto-upload helper that, above a size threshold, uploads a { type: 'data' } input via the provider's Files API and swaps in the handle — the large-input path #907 called out. fal already does a form of this internally.
Related: server-side URL buffering audit (#907 class)
Sweeping the repo for adapters that fetch + buffer image URL inputs locally:
- Gemini image — was buffering; fixed in #907 (now
fileDatapassthrough). - Gemini Veo (
imagePartToVeoImage) — still buffers HTTP URLs, but required: the predict API'sImagetype only acceptsimageBytesorgcsUri, no URI passthrough. - OpenAI image edits (
imagePartToFile) — still buffers, but required:/v1/images/editsis multipart and needs real bytes; there's no "fetch this URL" option. Afiles.uploadhandle would be the memory-safe alternative here. - Everyone else passes URLs through without buffering: Gemini chat, fal image, Grok image, Mistral vision, OpenAI vision. (OpenAI/Grok/
arrayBufferToBase64hits elsewhere are all output handling — downloading generated media — not input buffering.)
Scope / open questions
- Capability gating: which models/endpoints accept file handles varies (Imagen vs native Gemini image, Responses vs Chat Completions). Needs
feature-support/model-metawiring. - E2E: aimock coverage for upload endpoints (likely unit-first, mirroring the image-gen mapping note in
feature-support.ts). - Not all provider file handles are cross-compatible with all generation types — the type layer should reflect that.
References
- #907 — Gemini image URL passthrough (the immediate trigger)
- Gap analysis notes Files API as a broader Gemini gap (
.agent/gap-analysis/)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.