cloudflare / cloudflare/ai

`@cloudflare/tanstack-ai`: non-stream binding fetch discards OpenAI-shaped `binding.run` responses — structured output fails with "expected object, received string"

Open
#628 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

## Summary

With `@cloudflare/tanstack-ai` 0.2.1, a native Workers AI model, and the binding transport, `chat({ outputSchema })` fails with a misleading validation error:

```
Error: Validation failed: Invalid input: expected object, received string
code: 'structured-output-validation-failed'
```

The model is not at fault — it returns a valid, schema-conformant object. The adapter's **non-stream response wrapper** discards that output, because `binding.run` returns an OpenAI-shaped `chat.completion` body and the wrapper only understands the native `{ response: ... }` shape. This is a regression against the equivalent `workers-ai-provider` + AI SDK setup, whose response extraction handles both shapes.

## Reproducer

```ts
import { chat } from "@tanstack/ai";
import { createWorkersAiChat } from "@cloudflare/tanstack-ai";
import { z } from "zod";

const adapter = createWorkersAiChat("@cf/meta/llama-4-scout-17b-16e-instruct", {
binding: env.AI,
gateway: env.CF_AIG_ID,
});

await chat({
adapter,
systemPrompts: [systemPrompt],
messages: [{ role: "user", content: userMessage }],
outputSchema: z.object({ id: z.string().nullable() }),
});
```

## Actual

```
Error: Validation failed: Invalid input: expected object, received string
code: 'structured-output-validation-failed'
at runAgenticStructuredOutput (@tanstack/ai/dist/esm/activities/chat/index.js:1671)
cause: StandardSchemaValidationError
at parseWithStandardSchema (@tanstack/ai/.../tools/schema-converter.js:157)
at TextEngine.runStructuredFinalization (@tanstack/ai/.../chat/index.js:1303)
```

## Root cause

`wrapWorkersAiResultAsOpenAI` (`packages/tanstack-ai/src/utils/create-fetcher.ts`), on the non-stream path of `createWorkersAiBindingFetch`:

```ts
const responseObj = typeof result === "object" && result !== null ? result : { response: String(result) };
const message = {
role: "assistant",
content: typeof responseObj.response === "string" ? responseObj.response : /* … */ "",
};
```

The wrapper only understands the native `{ response: ... }` shape. But `binding.run` for this model (messages + `response_format`) returns an **OpenAI-shaped `chat.completion` body** — `{"choices":[{"message":{"content":"{\"id\":\"A1\"}"}}], "object":"chat.completion", …}` (vLLM passthrough fields like `routed_experts` included). That body has no `.response` key, so:

1. `content` becomes `""` — the model's valid constrained output is discarded
2. in `structuredOutput`, `JSON.parse("")` throws → `catch { data = rawText }` → `data = ""`, a string
3. → `Validation failed: Invalid input: expected object, received string`

The failure is deterministic for any model/route whose run endpoint returns the OpenAI shape.

Notably, the **streaming** transformer in the same file already handles this exact case:

```ts
if (parsed.choices !== undefined) { isOpenAiFormat = true; /* pass through */ }
```

The non-stream wrapper just never got the equivalent. It's the mirror image of #543 (native shape reaching a consumer that expects OpenAI shape — here an OpenAI shape reaching a wrapper that expects native).

**Ruled out:** the `{ name, strict, schema }` envelope the adapter sends as `response_format.json_schema`. Tested live via the AI Gateway `workers-ai/run/` REST endpoint and via `binding.run` directly — both the envelope and a bare JSON Schema engage constrained decoding correctly on this model, and the sibling `workers-ai-provider` distinction from #559 is not the issue here.

## A second defect: the silent parse fallback

[`adapters/workers-ai.ts` → `structuredOutput`](https://github.com/cloudflare/ai/blob/main/packages/tanstack-ai/src/adapters/workers-ai.ts) does:

```ts
try {
data = JSON.parse(rawText);
} catch {
data = rawText; // ← a failed parse becomes the raw string, with no signal
}
```

This is what turned an empty-content bug into a confusing type error about the user's schema. Whatever happens upstream of it, a non-JSON reply should not silently flow onward as a `string` — surfacing `rawText` in the error would have made the root cause obvious from the first stack trace.

## Fix (verified locally)

Pass OpenAI-shaped bodies through untouched, mirroring the stream transformer (patched `dist` locally; `chat({ outputSchema })` then returns the validated object against the live model):

```ts
function wrapWorkersAiResultAsOpenAI(result, model, salvageContext) {
if (typeof result === "object" && result !== null && Array.isArray(result.choices)) {
return new Response(JSON.stringify(result), { headers: { "Content-Type": "application/json" } });
}
// …existing native-shape wrapping…
}
```

Happy to turn this into a PR if useful.

## Environment

- `@cloudflare/tanstack-ai`: 0.2.1
- `@tanstack/ai`: 0.42.0
- Model: `@cf/meta/llama-4-scout-17b-16e-instruct`
- Transport: direct binding + gateway id (`{ binding: env.AI, gateway: env.CF_AIG_ID }`)
- Runtime: Cloudflare Workers

## Related

- #543 — the same shape mismatch on the gateway-binding path, in the opposite direction

Contributor guide

Open the contributing guide

Research direction

Read packages/tanstack-ai/src/utils/create-fetcher.ts, especially the non-stream path of createWorkersAiBindingFetch, and compare it with the streaming transformer’s OpenAI-shape handling. Run the provided chat({ outputSchema }) reproducer with the Workers AI binding; done means OpenAI-shaped binding.run responses preserve structured output, native responses still work, and the silent parse fallback in adapters/workers-ai.ts has been assessed.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.