ByteDance-Seed / ByteDance-Seed/DAComp

high RPM

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
441
Forks
5
PR merge metrics
No merged PRs in 30d

Description

## Excessive retry loop in `_http_completion` causes high RPM during transient errors

### Location

`dacomp-da/evaluation_suite/core/llm_client.py`, line 157 (function `_http_completion`)

### Problem

The HTTP completion function uses a retry loop of **3000 iterations with 0.2s sleep**, which causes two issues:

1. **Excessive RPM during transient errors**: When the API gateway returns a 502/503 or other non-JSON response, the code retries at ~5 requests/second (300 RPM). This can overwhelm the gateway, trigger additional rate limiting, and exacerbate the outage.

2. **Ineffective for persistent failures**: If the model consistently returns invalid content (e.g., truncated JSON), retrying 3000 times won't fix it — it just burns 10 minutes of wall time and thousands of wasted API calls.

### Current behavior

```python
for attempt in range(1, 3001):
# ... request ...
except ValueError:
# Non-JSON response (e.g., 502 HTML page)
time.sleep(0.2) # 300 RPM retry rate
continue

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.