cloudflare / cloudflare/ai

`@cloudflare/tanstack-ai`: Workers AI gateway-binding mode returns native response shape, breaks structured output (and streaming)

Open
#543 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

## Summary

When `createWorkersAiChat` is configured with `binding: env.AI.gateway(id)` (the gateway-binding mode shown in the README), the adapter routes through the AI Gateway's universal-endpoint `run/` path and forwards Workers AI's **native** response shape (`{ response, tool_calls, usage }`) unchanged to the OpenAI SDK. The SDK expects `{ choices: [...] }`, so `structuredOutput` throws and streaming text presumably also fails to parse content.

Direct binding mode (`binding: env.AI`) works because `createWorkersAiBindingFetch` explicitly translates the native shape into OpenAI shape before the SDK sees it; the same translation is missing from `createGatewayFetch` for the `workers-ai` branch.

## Reproducer

```ts
import { chat } from "@tanstack/ai";
import { createWorkersAiChat } from "@cloudflare/tanstack-ai";
import { z } from "zod";

const adapter = createWorkersAiChat("@cf/meta/llama-4-scout-17b-16e-instruct", {
binding: env.AI.gateway(env.CF_AIG_ID),
});

await chat({
adapter,
systemPrompts: ["..."],
messages: [{ role: "user", content: "..." }],
outputSchema: z.object({ name: z.string().nullable() }),
});
```

Throws:

```
Error: Workers AI structured output returned no choices:
{"response":{"name":"..."},"tool_calls":[],"usage":{"prompt_tokens":518,"completion_tokens":11,"total_tokens":529,"prompt_tokens_details":{"cached_tokens":0}}}
at WorkersAiTextAdapter.structuredOutput (.../workers-ai.mjs)
```

The model actually returned a perfectly valid `{ name: "..." }` — the adapter just can't unwrap the universal-endpoint response.

## Root cause

In [`create-fetcher.ts` → `createGatewayFetch`](https://github.com/cloudflare/ai/blob/main/packages/tanstack-ai/src/utils/create-fetcher.ts), the `workers-ai` branch rewrites the endpoint to `run/`:

```ts
if (provider === "workers-ai") {
if (!request.endpoint.startsWith("run/")) {
request.endpoint = `run/${query.model}`;
}
delete query.model;
delete query.instructions;
}
```

The gateway forwards this to Workers AI's native `/ai/run/` API, which returns `{ response, tool_calls, usage }`. That payload is then passed straight to the OpenAI SDK, which expects `{ choices: [...] }`.

`createWorkersAiBindingFetch` (used in direct-binding mode) does the translation here:

```ts
const responseObj = typeof result === "object" && result !== null ? result : { response: String(result) };
const message = {
role: "assistant",
content: typeof responseObj.response === "string"
? responseObj.response
: typeof responseObj.response === "object" && responseObj.response !== null
? JSON.stringify(responseObj.response)
: "",
};
// ... wrapped into { choices: [{ index: 0, message, finish_reason }] }
```

The same translation is needed in the gateway path, or the gateway path should hit the OpenAI-compat route at `workers-ai/v1/chat/completions` instead of the universal `run/` endpoint.

## Workaround

Use direct binding mode (`{ binding: env.AI }`) — loses AI Gateway caching / observability for Workers AI calls.

## Versions

- `@cloudflare/tanstack-ai` 0.1.9 (latest)
- `@tanstack/ai` 0.18.0
- Verified broken in every release of `@cloudflare/tanstack-ai` that ships the gateway-binding path (0.1.7 → 0.1.9)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.