cloudflare / cloudflare/ai

workers-ai-provider: binding streams never honor abortSignal, so AI SDK timeout/abortSignal hang forever on stalled streams

Open
#640 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

## Summary

When streaming through the Workers AI **binding** (`createWorkersAI({ binding: env.AI })`), the provider does not interrupt a pending read on the binding's `ReadableStream` when the request's `AbortSignal` fires. Since AI SDK v7 enforces `streamText({ timeout })` and `abortSignal` **solely** by aborting the signal it passes to `doStream` (there is no SDK-side read race), a stream that stalls without closing hangs the consumer forever — `timeout: { chunkMs, firstChunkMs, totalMs }` and caller-side `AbortController.abort()` are all silently ineffective.

This matters in practice because Workers AI streams do occasionally stall mid-generation without closing (we've observed this in production), which is exactly the situation timeouts exist for.

## Versions

- `workers-ai-provider` 4.0.0 (also reproduces on 3.3.1 with ai v6)
- `ai` 7.0.66
- Node 24 / workerd — reproduces in both

## Minimal repro

```js
import { streamText } from "ai";
import { createWorkersAI } from "workers-ai-provider";

const enc = new TextEncoder();
// Binding whose stream emits one chunk, then stalls (never closes) —
// mimics a stalled Workers AI stream.
const binding = {
run: async () =>
new ReadableStream({
start(c) {
c.enqueue(enc.encode(`data: ${JSON.stringify({ response: "Hello" })}\n\n`));
// deliberately no close(), no further chunks
},
}),
};

const model = createWorkersAI({ binding })("@cf/meta/llama-3.3-70b-instruct-fp8-fast");

const result = streamText({
model,
prompt: "hi",
timeout: { firstChunkMs: 500, chunkMs: 500, totalMs: 2000 },
});
for await (const t of result.textStream) console.log("token:", t);
console.log("done"); // never reached — process hangs (unsettled await)
```

Expected: the stream ends (SDK abort semantics) within ~500 ms and `onAbort` fires / result promises reject with `TimeoutError`.

Actual: hangs forever. Same behavior when using `abortSignal` with a manual `AbortController` instead of `timeout`.

## Root cause

`doStream` forwards the signal to the binding (`this.config.binding.run(model, inputs, { signal: options.abortSignal })`), but once the `ReadableStream` is returned, the pipeline in `getMappedStream` (`rawStream → SSEDecoder → TransformStream`) never observes the signal. A `reader.read()` that is pending when the signal fires is never rejected or cancelled, so the mapped stream never terminates. Fetch-based providers get abort-on-read for free from `fetch`; the binding path does not.

## Suggested fix

Race each read against the signal and error the mapped stream with `signal.reason` (which preserves the SDK's `TimeoutError`/`AbortError` semantics). We're running this in production as a `patch-package` patch:

```js
function raceAbort(stream, signal) {
if (!signal) return stream;
const reader = stream.getReader();
const aborted = new Promise((_, reject) => {
const fail = () => reject(signal.reason ?? new DOMException("Aborted", "AbortError"));
if (signal.aborted) fail();
else signal.addEventListener("abort", fail, { once: true });
});
aborted.catch(() => {});
return new ReadableStream({
async pull(controller) {
try {
const { done, value } = await Promise.race([reader.read(), aborted]);
if (done) controller.close();
else controller.enqueue(value);
} catch (e) {
reader.cancel(e).catch(() => {});
controller.error(e);
}
},
cancel(reason) {
return reader.cancel(reason);
},
});
}
```

applied at the `doStream` call site:

```js
getMappedStream(raceAbort(response, options.abortSignal), { ... })
```

With this in place, `streamText`'s `timeout` options and `abortSignal` behave per the SDK contract: the text stream ends cleanly, `onAbort` fires, and result promises reject with `TimeoutError`. Verified with unit tests against stalled fake streams and against live Workers AI (a 1 ms `firstChunkMs` terminates a real stream in ~200 ms).

Happy to turn this into a PR if useful.

Contributor guide

Open the contributing guide

Research direction

Start at the binding path in doStream and follow the response into getMappedStream, focusing on how pending ReadableStream reads react to options.abortSignal. Reproduce the stalled fake stream from the issue, then add unit coverage for timeout and manual abort behavior; done means the mapped stream terminates and preserves signal.reason, including TimeoutError or AbortError.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.