cloudflare / cloudflare/ai

AI Gateway doesn't track costs/tokens for streaming Azure OpenAI responses

Open
#470 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

AI Gateway correctly tracks costs and token usage for non-streaming Azure OpenAI requests, but fails to do so for streaming (SSE) responses.

Observed behavior:
- Non-streaming (stream: false): Model shows as gpt-5.3-chat with correct token counts and costs
- Streaming (stream: true): Model shows as gpt-5.3 with no token counts or costs

The streaming response DOES include usage data in the final SSE chunk (verified by logging the raw response):
data: {"choices":[],"model":"gpt-5.3-chat-2026-03-03","usage":{"completion_tokens":137,"prompt_tokens":3886,"total_tokens":4023}}
data: [DONE]

The stream_options: { include_usage: true } flag is set in the request body, and Azure returns the usage correctly — Gateway just doesn't read it.

For Cloudflare employees, an example request ID is 01KMY0CE9AKNBRFBHB7J9W4X2G

Contributor guide

Open the contributing guide

Research direction

Start by tracing the AI Gateway's Azure OpenAI streaming (SSE) response handling and compare it with the non-streaming path. Use the final SSE chunk shown in the issue as the expected usage source; done means streaming responses report the model, token counts, and costs consistently with non-streaming responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, typescript
Domain
api, backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.