AI Gateway doesn't track costs/tokens for streaming Azure OpenAI responses
- Dominant language
- TypeScript
- Stars
- 1.2k
- Forks
- 345
- Avg merge
- 13h 31m
- Merged PRs (30d)
- 1
Description
AI Gateway correctly tracks costs and token usage for non-streaming Azure OpenAI requests, but fails to do so for streaming (SSE) responses.
Observed behavior:
- Non-streaming (stream: false): Model shows as gpt-5.3-chat with correct token counts and costs
- Streaming (stream: true): Model shows as gpt-5.3 with no token counts or costs
The streaming response DOES include usage data in the final SSE chunk (verified by logging the raw response):
data: {"choices":[],"model":"gpt-5.3-chat-2026-03-03","usage":{"completion_tokens":137,"prompt_tokens":3886,"total_tokens":4023}}
data: [DONE]
The stream_options: { include_usage: true } flag is set in the request body, and Azure returns the usage correctly — Gateway just doesn't read it.
For Cloudflare employees, an example request ID is 01KMY0CE9AKNBRFBHB7J9W4X2G
Contributor guide
Research direction
Start by tracing the AI Gateway's Azure OpenAI streaming (SSE) response handling and compare it with the non-streaming path. Use the final SSE chunk shown in the issue as the expected usage source; done means streaming responses report the model, token counts, and costs consistently with non-streaming responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, typescript
- Domain
- api, backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100