openai / openai/openai-python

Inconcistency between Responses API and Chat Completions API on rate limit errors

Open
#2,699 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
31.6k
Forks
5.7k
Avg merge
1d 6h
Merged PRs (30d)
96

Description

Confirm this is an issue with the Python library and not an underlying OpenAI API
  • This is an issue with the Python library
Describe the bug

I noticed that rate limit errors are raised differently in the two APIs during streaming. In chat completions the error is raised and retried by the client itself on the initial call, before reading the stream. On the other hand, the first call to responses.create(...) returns a stream object successfully and then upon reading the stream, an APIError is raised indicating a rate limit. As a result, the whole retrying logic in the client is completely skipped and the user has to implement their own. It is also unclear whether an error could be raised mid-stream which adds to the complexity of the fix.

After inspecting the client's code, I initially thought that this is not a library issue but a misalignment with the model's service. I contacted Microsoft support since they host our models but they insisted I raise an issue here.

To Reproduce

To reproduce this you would need an Azure hosted OpenAI model ( I have personally already tried this with gpt-4o and gpt-5 ). Lower your token rate limit threshold to a low enough value and use the below snippet.

Code snippets
from openai import OpenAI

client = OpenAI(  
  base_url = "your-endpoint-v1",
  api_key="your-api-key"
)

response = client.responses.create(
    model="your-model",
    store=False,
    stream=True,
    input='a long enough prompt'*10000 # to hit a rate limit
)

for event in response:
   continue # an APIError is raised here
OS

Linux, Windows

Python version

Python >=3.11.9

Library version

openai v1.109.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the responses.create(..., stream=True) entry point and compare its rate-limit error handling with the Chat Completions streaming path, especially the client retry logic before iteration. Reproduce against an Azure-hosted model with a long input, then determine and test consistent behavior for errors raised when creating the stream and while reading it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.