`@cloudflare/tanstack-ai`: Workers AI gateway-binding mode returns native response shape, breaks structured output (and streaming)
- Dominant language
- TypeScript
- Stars
- 1.2k
- Forks
- 345
- Avg merge
- 13h 31m
- Merged PRs (30d)
- 1
Description
## Summary
When `createWorkersAiChat` is configured with `binding: env.AI.gateway(id)` (the gateway-binding mode shown in the README), the adapter routes through the AI Gateway's universal-endpoint `run/` path and forwards Workers AI's **native** response shape (`{ response, tool_calls, usage }`) unchanged to the OpenAI SDK. The SDK expects `{ choices: [...] }`, so `structuredOutput` throws and streaming text presumably also fails to parse content.
Direct binding mode (`binding: env.AI`) works because `createWorkersAiBindingFetch` explicitly translates the native shape into OpenAI shape before the SDK sees it; the same translation is missing from `createGatewayFetch` for the `workers-ai` branch.
## Reproducer
```ts
import { chat } from "@tanstack/ai";
import { createWorkersAiChat } from "@cloudflare/tanstack-ai";
import { z } from "zod";
const adapter = createWorkersAiChat("@cf/meta/llama-4-scout-17b-16e-instruct", {
binding: env.AI.gateway(env.CF_AIG_ID),
});
await chat({
adapter,
systemPrompts: ["..."],
messages: [{ role: "user", content: "..." }],
outputSchema: z.object({ name: z.string().nullable() }),
});
```
Throws:
```
Error: Workers AI structured output returned no choices:
{"response":{"name":"..."},"tool_calls":[],"usage":{"prompt_tokens":518,"completion_tokens":11,"total_tokens":529,"prompt_tokens_details":{"cached_tokens":0}}}
at WorkersAiTextAdapter.structuredOutput (.../workers-ai.mjs)
```
The model actually returned a perfectly valid `{ name: "..." }` — the adapter just can't unwrap the universal-endpoint response.
## Root cause
In [`create-fetcher.ts` → `createGatewayFetch`](https://github.com/cloudflare/ai/blob/main/packages/tanstack-ai/src/utils/create-fetcher.ts), the `workers-ai` branch rewrites the endpoint to `run/`:
```ts
if (provider === "workers-ai") {
if (!request.endpoint.startsWith("run/")) {
request.endpoint = `run/${query.model}`;
}
delete query.model;
delete query.instructions;
}
```
The gateway forwards this to Workers AI's native `/ai/run/` API, which returns `{ response, tool_calls, usage }`. That payload is then passed straight to the OpenAI SDK, which expects `{ choices: [...] }`.
`createWorkersAiBindingFetch` (used in direct-binding mode) does the translation here:
```ts
const responseObj = typeof result === "object" && result !== null ? result : { response: String(result) };
const message = {
role: "assistant",
content: typeof responseObj.response === "string"
? responseObj.response
: typeof responseObj.response === "object" && responseObj.response !== null
? JSON.stringify(responseObj.response)
: "",
};
// ... wrapped into { choices: [{ index: 0, message, finish_reason }] }
```
The same translation is needed in the gateway path, or the gateway path should hit the OpenAI-compat route at `workers-ai/v1/chat/completions` instead of the universal `run/` endpoint.
## Workaround
Use direct binding mode (`{ binding: env.AI }`) — loses AI Gateway caching / observability for Workers AI calls.
## Versions
- `@cloudflare/tanstack-ai` 0.1.9 (latest)
- `@tanstack/ai` 0.18.0
- Verified broken in every release of `@cloudflare/tanstack-ai` that ships the gateway-binding path (0.1.7 → 0.1.9)
Contributor guide
Assessment
This issue has not been assessed yet.