openai / openai/openai-agents-python

BackendSpanExporter drops the whole batch on HTTP 429 instead of retrying with Retry-After

Open
#5,023 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
29.6k
Forks
4.8k
Avg merge
1d 20h
Merged PRs (30d)
123

Description

Please read this first
  • Have you read the docs? Yes: Tracing and the BackendSpanExporter reference.
  • Have you searched for related issues? Yes. #4683 / #4711 cover retries being disabled after shutdown, #4983 covers a non-serializable value dropping a batch, #5008 covers the endpoint. None covers how the exporter classifies HTTP status codes.
Describe the bug

BackendSpanExporter._export_with_deadline() treats every 4xx response as final:

# If the response is a client error (4xx), we won't retry
if 400 <= response.status_code < 500:
    logger.error("[non-fatal] Tracing client error %s: %s", ...)
    break

That includes 429 Too Many Requests. A single rate-limited response from /v1/traces/ingest discards the whole batch, up to max_batch_size (128) traces and spans, with no retry and without looking at Retry-After. The run itself succeeds, so the only symptom is a gap in the dashboard and one [non-fatal] log line. The same applies to 408 and 409, which the OpenAI Python client (_base_client._should_retry) retries together with 429 and 5xx; the exporter only mirrors the 5xx half of that policy.

The export path already has everything needed to retry: max_retries, base_delay, max_delay, the backoff loop, and agents.models._retry_runtime.parse_retry_after_value / parse_retry_after_ms for the header.

Debug information
  • Agents SDK version: main at fbf59a40 (also 0.22.0)
  • Python version: 3.10.12
  • Operating system: Linux
  • Model and model provider: none needed; httpx2.Client is patched
  • Does the issue reproduce with the latest Agents SDK release? Yes
  • Does the issue occur consistently or intermittently? Consistently for any 429
ERROR:openai.agents:[non-fatal] Tracing client error 429. Response data is redacted.
post calls: 1
Repro steps
import logging
from unittest.mock import MagicMock, patch

from agents.tracing.processors import BackendSpanExporter
from agents.tracing.spans import SpanImpl
from agents.tracing.span_data import CustomSpanData

logging.basicConfig(level=logging.WARNING)

with patch("httpx2.Client") as client:
    limited = MagicMock(status_code=429, headers={"retry-after": "1"})
    limited.text = '{"error": {"message": "Rate limit reached"}}'
    ok = MagicMock(status_code=200)
    client.return_value.post.side_effect = [limited, ok]

    exporter = BackendSpanExporter(api_key="sk-test", max_retries=3, base_delay=0.1)
    span = SpanImpl(
        trace_id="trace_1",
        span_id="span_1",
        parent_id=None,
        processor=MagicMock(),
        span_data=CustomSpanData(name="probe", data={}),
        tracing_api_key=None,
    )
    exporter.export([span])
    print("post calls:", client.return_value.post.call_count)  # 1: the batch is gone
Expected behavior

408, 409 and 429 are retried with the existing backoff up to max_retries, waiting at least the advertised Retry-After / retry-after-ms (bounded by max_delay), the way the OpenAI client and the SDK's own model retry helpers already treat these statuses. Other 4xx responses stay non-retried. In the script above the second call succeeds and post calls is 2.

A fix with regression tests is ready and I will attach it right away.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at BackendSpanExporter._export_with_deadline() and compare its status handling with agents.models._retry_runtime.parse_retry_after_value, parse_retry_after_ms, and the OpenAI client's _base_client._should_retry. Add regression coverage for 408, 409, and 429, including Retry-After, retry-after-ms, max_delay, and preserving non-retry behavior for other 4xx; the supplied repro should make the second post succeed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.