ByteDance-Seed / ByteDance-Seed/DAComp
high RPM
- Dominant language
- Python
- Stars
- 441
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
## Excessive retry loop in `_http_completion` causes high RPM during transient errors
### Location
`dacomp-da/evaluation_suite/core/llm_client.py`, line 157 (function `_http_completion`)
### Problem
The HTTP completion function uses a retry loop of **3000 iterations with 0.2s sleep**, which causes two issues:
1. **Excessive RPM during transient errors**: When the API gateway returns a 502/503 or other non-JSON response, the code retries at ~5 requests/second (300 RPM). This can overwhelm the gateway, trigger additional rate limiting, and exacerbate the outage.
2. **Ineffective for persistent failures**: If the model consistently returns invalid content (e.g., truncated JSON), retrying 3000 times won't fix it — it just burns 10 minutes of wall time and thousands of wasted API calls.
### Current behavior
```python
for attempt in range(1, 3001):
# ... request ...
except ValueError:
# Non-JSON response (e.g., 502 HTML page)
time.sleep(0.2) # 300 RPM retry rate
continue
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.