openai / openai/openai-openapi

[Streaming] Token usage not returned when stream is aborted mid-generation

Open
#539 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

api enhancement
Dominant language
No language data
Stars
2.5k
Forks
527
Avg merge
1h 46m
Merged PRs (30d)
2

Description

Summary

When using streaming with stream_options: { include_usage: true },
the token usage chunk is never received if the stream is aborted
mid-generation. This makes accurate token tracking impossible for
apps that allow users to cancel responses early.

Current Behavior

The usage chunk is emitted as the very last SSE event, after generation
fully completes:

data: {"choices": [{"delta": {"content": "..."}}]} ← content chunks
data: {"choices": [{"finish_reason": "stop"}]} ← generation done
data: {"usage": {"prompt_tokens": 10, ...}} ← never received on abort
data: [DONE]

If the client aborts at any point before generation finishes, the usage
chunk is never sent.

Expected Behavior

One of the following:

Option A — Emit a partial usage chunk when server detects client disconnect:
data: {"usage": {"prompt_tokens": 42, "completion_tokens": 17, "partial": true}}

Option B — Include a running completion_tokens_so_far counter
on every chunk (opt-in via stream_options):
data: {"delta": {"content": "..."}, "usage": {"completion_tokens_so_far": 5}}

Why It Matters

  • Billing per user becomes inaccurate when generation is cancelled
  • Common real-world patterns that trigger this: user closes tab,
    presses stop button, request timeout fires
  • Workarounds like client-side tiktoken counting are approximate
    and inconsistent across models

Reproduction

const stream = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Write a very long essay" }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  // abort after first chunk
  stream.controller.abort();
  break;
}
// usage is never received

Workaround Today

Client-side token estimation using js-tiktoken — inaccurate,
especially for tokens with special characters or multi-byte sequences.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the OpenAPI definitions for chat completion streaming and the documented stream_options behavior described in this issue. Determine which proposed abort-time usage behavior should be specified, then update the API contract and verify that the resulting schema and streaming semantics cover the provided JavaScript reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, openapi
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.