cloudflare / cloudflare/ai

Workers AI OpenAI-compatible streaming: `delta.content` sometimes arrives as a JSON number instead of a string

Open
#651 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

**Endpoint:** `POST https://api.cloudflare.com/client/v4/accounts//ai/v1/chat/completions` with `stream: true`
**Model:** `@cf/meta/llama-3.3-70b-instruct-fp8-fast`
**Seen:** 2 September 2026

I hit this running [recourse](https://github.com/ibrahimhajjaj/recourse), a customer support agent that learns your own content, against this endpoint from a Worker.

Some streaming chunks carry `choices[0].delta.content` as a JSON number rather than a string. Here is one, copied out of my Worker's logs exactly as it arrived:

```json
{"id":"id-1788378160014","created":1788378160,"model":"@cf/meta/llama-3.3-70b-instruct-fp8-fast","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":6}}],"usage":{"prompt_tokens":0,"completion_tokens":1,"total_tokens":1,"prompt_tokens_details":{"cached_tokens":0},"neurons":0.20480532944202423}}
```

The OpenAI streaming format has that field as a string, so a client that checks it against the spec rejects the chunk.

## What it looks like from the outside

Answers quietly lose characters, and then stop. A citation marker `[1]` comes out as `[]`, because the `1` is its own token. A price of `$0.005` comes out as `$`. After a few of those the stream dies part way through a sentence.

It took me a while to work out this was not my bug, because the visible symptom is missing text rather than an error.

## What I think is going on

The token text looks like it goes through a JSON parse somewhere and gets re-serialised as whatever it parsed to:

```
"6" -> 6
"0.005" -> 0.005
"155724\n" -> 155724 (the newline is eaten)
"005" -> parse fails, so it stays "005"
"$0.005" -> parse fails, so it stays "$0.005"
```

That accounts for every value I have seen, and it explains why only some numeric-looking text breaks while the rest is fine.

## Probably the same defect as #277

#277 reports AutoRAG streaming emitting numeric JSON for `response` chunks on this same model, with the values `155724` and `3e+50` and lost newlines. Different field, different surface, same coercion. That one has been open since September 2025. If the parse sits upstream of the OpenAI-compatible mapping, one fix covers both.

## Who this hits

- **Vercel AI SDK** through `@ai-sdk/openai-compatible`: throws `AI_TypeValidationError`, `expected "string"`, `path ["choices",0,"delta","content"]`, and the answer stops mid-stream.
- **Your own `workers-ai-provider@4.0.0`**: does not throw. It tests `textDelta.length > 0`, which is `undefined > 0` for a number, so the token is dropped without a word. Silent character loss rather than a crash.

## Reproducing

Stream any completion whose text contains a standalone number. Prompts about prices or numbered references hit it most often. The bad frames are intermittent, so it can take a few turns.

```bash
curl -N "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/v1/chat/completions" \
-H "Authorization: Bearer $CLOUDFLARE_AI_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
"stream": true,
"messages": [{"role": "user", "content": "Reply with exactly: [1] the price is $0.005 a minute."}]
}' | grep '"content":[^"]'
```

Any line that comes back has an unquoted value in `content`.

## What I am doing until it is fixed

Every client on this endpoint has to put the value back before its own validation runs, so I ended up shipping a small `fetch` wrapper that rewrites `delta.content` to a string whenever it arrives as a number or a boolean: [`repairNumericContent`](https://github.com/ibrahimhajjaj/recourse/blob/4deb6ec/packages/core/src/models.ts#L157-L229). That keeps the digits and stops the stream dying, and it is safe precisely because a compliant server never puts a number there.

It does nothing for the whitespace, though. By the time the chunk reaches me the newline after `155724` is already gone, so that half is only fixable at your end.

## Expected

`delta.content` is always a string, and the token text reaches the wire as it left the tokeniser, whitespace included.

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied curl reproduction against the chat completions endpoint and inspect the streamed chunks for unquoted numeric content. Compare the behavior with related issue #277 and trace the OpenAI-compatible streaming response path. Done means delta.content is always a string and numeric-looking token text, including whitespace, is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.