Hang on 429 response from OpenAI

I was testing out the new(ish) project-based request limits, and I noticed that when I use llm prompt in the default (streaming) mode, it hangs until the request limit resets (probably the same is true of TPM limit but I haven't tested it). If I include the logging I can see the response has a 429 status, so it probably should abort instead, or at least log to stderr that it hit the rate limit so you know what's happening.

It hangs for about 50 seconds on Response: Body:

LLM_OPENAI_SHOW_RESPONSES=true llm -m gpt-4 --key oai-playground 'hello world'
Request: POST https://api.openai.com/v1/chat/completions
  Headers:
    host: api.openai.com
    connection: keep-alive
    accept: application/json
    content-type: application/json
    user-agent: OpenAI/Python 1.14.2
    x-stainless-lang: python
    x-stainless-package-version: 1.14.2
    x-stainless-os: MacOS
    x-stainless-arch: arm64
    x-stainless-runtime: CPython
    x-stainless-runtime-version: 3.12.2
    authorization: [...]
    x-stainless-async: false
    content-length: 92
  Body:
    {
      "messages": [
        {
          "role": "user",
          "content": "hello world"
        }
      ],
      "model": "gpt-4",
      "stream": true
    }
Response: status_code=429
  Headers:
    date: Tue, 28 May 2024 18:45:25 GMT
    content-type: application/json; charset=utf-8
    content-length: 400
    connection: keep-alive
    retry-after: 60
    retry-after-ms: 60000
    vary: Origin
    x-ratelimit-limit-project-requests: 1
    x-ratelimit-limit-project-tokens: 1000
    x-ratelimit-limit-requests: 10000
    x-ratelimit-limit-tokens: 300000
    x-ratelimit-remaining-project-requests: 0
    x-ratelimit-remaining-project-tokens: 980
    x-ratelimit-remaining-requests: 9999
    x-ratelimit-remaining-tokens: 299980
    x-ratelimit-reset-project-requests: 50.708s
    x-ratelimit-reset-project-tokens: 1.2s
    x-ratelimit-reset-requests: 6ms
    x-ratelimit-reset-tokens: 4ms
    x-request-id: req_67a765e2385da58bd8e42e27e25b43cb
    strict-transport-security: max-age=15724800; includeSubDomains
    cf-cache-status: DYNAMIC
    set-cookie: __cf_bm=...
    server: cloudflare
    cf-ray: 88b066d1f866c38d-SEA
    alt-svc: h3=":443"; ma=86400
  Body:
Request: POST https://api.openai.com/v1/chat/completions
  Headers:
    host: api.openai.com
    connection: keep-alive
    accept: application/json
    content-type: application/json
    user-agent: OpenAI/Python 1.14.2
    x-stainless-lang: python
    x-stainless-package-version: 1.14.2
    x-stainless-os: MacOS
    x-stainless-arch: arm64
    x-stainless-runtime: CPython
    x-stainless-runtime-version: 3.12.2
    authorization: [...]
    x-stainless-async: false
    cookie: __cf_bm=...
    content-length: 92
  Body:
    {
      "messages": [
        {
          "role": "user",
          "content": "hello world"
        }
      ],
      "model": "gpt-4",
      "stream": true
    }
Response: status_code=200
  Headers:
    date: Tue, 28 May 2024 18:46:26 GMT
    content-type: text/event-stream; charset=utf-8
    transfer-encoding: chunked
    connection: keep-alive
    openai-organization: wunderkind-zumhsi
    openai-processing-ms: 308
    openai-version: 2020-10-01
    strict-transport-security: max-age=15724800; includeSubDomains
    x-ratelimit-limit-project-requests: 1
    x-ratelimit-limit-project-tokens: 1000
    x-ratelimit-limit-requests: 10000
    x-ratelimit-limit-tokens: 300000
    x-ratelimit-remaining-project-requests: 0
    x-ratelimit-remaining-project-tokens: 980
    x-ratelimit-remaining-requests: 9999
    x-ratelimit-remaining-tokens: 299980
    x-ratelimit-reset-project-requests: 1m0s
    x-ratelimit-reset-project-tokens: 1.2s
    x-ratelimit-reset-requests: 6ms
    x-ratelimit-reset-tokens: 4ms
    x-request-id: req_2f2037b9050d81e1aff8fb679de2b08d
    cf-cache-status: DYNAMIC
    server: cloudflare
    cf-ray: 88b0684a0c31767c-SEA
    alt-svc: h3=":443"; ma=86400
  Body:
data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,
"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}]}


Hellodata: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"!"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" How"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" can"},"logprobs":null,"finish_reason":null}]}


! How candata: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" I"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" assist"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" you"},"logprobs":null,"finish_reason":null}]}


 I assist youdata: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" today"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"?"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-9Tw81hPK1txeymuGqWYFVAcKNLIsw","object":"chat.completion.chunk","created":1716921985,"model":"gpt-4-0613","system_fingerprint":null,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}

data: [DONE]


 today?

Start by reproducing the issue with llm prompt in default streaming mode and the shown LLM_OPENAI_SHOW_RESPONSES=true command, using the OpenAI chat completions request. Trace how the CLI handles the logged 429 response; done means it exits or reports the rate limit promptly instead of waiting for the reset.