cloudflare / cloudflare/ai

[ai-gateway-provider] Concurrent requests can receive each other's responses

Open
#620 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

## Summary

When an `AiGatewayChatLanguageModel` is created once and reused for concurrent requests, a request can receive a response generated for another request. This can expose one user's model output to another user in applications that share a model instance.

Internally, `processModelRequest()` replaces each wrapped model's `config.fetch` with a request-local collector. Concurrent calls mutate the same shared `config.fetch`, causing one collector to capture multiple requests while another captures none. The resulting gateway responses can then be missing or returned to the wrong caller.

## Environment

- `ai-gateway-provider@3.2.0`
- `ai@6.0.208`
- Node.js 24.12.0

## Reproduction

```bash
mkdir ai-gateway-concurrency-repro
cd ai-gateway-concurrency-repro
npm init -y
npm install ai@6.0.208 ai-gateway-provider@3.2.0
```

Save as `repro.mjs`:

```js
import { generateText } from 'ai';
import { createAiGateway } from 'ai-gateway-provider';
import { createDeepSeek } from 'ai-gateway-provider/providers/deepseek';

const gateway = createAiGateway({
accountId: 'test',
gateway: 'test',
apiKey: 'test',
});

const deepseek = createDeepSeek();
const model = gateway(deepseek('deepseek-chat'));
const gatewayBodies = [];

globalThis.fetch = async (_url, init) => {
const body = JSON.parse(init.body);
const prompts = body.map(request => request.query?.messages?.at(-1)?.content);
gatewayBodies.push(prompts);

const prompt = prompts[0];
return new Response(
JSON.stringify({
id: 'id',
object: 'chat.completion',
created: 0,
model: 'deepseek-chat',
choices: [
{
index: 0,
message: { role: 'assistant', content: `ANSWER:${prompt}` },
finish_reason: 'stop',
},
],
usage: {
prompt_tokens: 1,
completion_tokens: 1,
total_tokens: 2,
},
}),
{
headers: {
'content-type': 'application/json',
'cf-aig-step': '0',
},
},
);
};

const ask = prompt => generateText({ model, prompt, maxRetries: 0 }).then(result => result.text);

const [a, b] = await Promise.all([ask('A'), ask('B')]);

console.log({ a, b, gatewayBodies });
```

Run:

```bash
node repro.mjs
```

Output:

```text
{
a: 'ANSWER:undefined',
b: 'ANSWER:A',
gatewayBodies: [ [], [ 'A', 'B' ] ]
}
```

The reproduction stubs only the final gateway `fetch`; no Cloudflare or provider credentials are required.

## Expected behavior

Concurrent calls using the same model remain isolated:

```text
{ a: 'ANSWER:A', b: 'ANSWER:B', gatewayBodies: [ [ 'A' ], [ 'B' ] ] }
```

## Actual behavior

One gateway request has an empty body and the other contains both provider requests. The responses are consequently missing or assigned to the wrong caller.

## Workaround

Create the provider model and gateway wrapper separately for each call instead of sharing the resulting model instance.

Contributor guide

Open the contributing guide

Research direction

Start at processModelRequest(), where the issue reports that each request replaces the wrapped model's config.fetch, and trace how concurrent calls collect gateway requests and responses. Reproduce the issue with the provided repro.mjs, then add coverage showing that shared-model calls keep their requests and responses isolated under Promise.all.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.