OpenHands / OpenHands/benchmarks
feat: respect registry rate limits
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
We should add a rate limit checks to building images to avoid overwhelming docker.
A generic 429 Too Many Requests error, or a specific "Too Many Requests (HAP429)" error is generally referring to the abuse limit, described here: https://docs.docker.com/docker-hub/usage/#abuse-rate-limit
This limit applies to all requests to Hub properties including web pages, APIs, and image pulls. The limit is applied per-IP, and while the limit changes over time depending on load and other factors, it's in the order of thousands of requests per minute. The abuse limit applies to all users equally regardless of account level.
To avoid hitting this limit accidentally we should try to keep the api calls and image pulls to under 100 a minute.
Assuming we use 100 workers, this shouldn't delay evaluations. Even if we use 500 workers - unlikely due to LLM rate limits - it would only delay it by 4 minutes. Keeping it under 100 prevents accidentally reaching the 1000 limit when running multiple evaluations in parallel
This feature affects SWE-Bench, commit0, and SWT-bench and all feature benchmarks.
GAIA implements its own rate limit logic until (https://github.com/OpenHands/benchmarks/issues/171) is implemented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the image-building, registry API, and image-pull entry points used by SWE-Bench, commit0, SWT-bench, and feature benchmarks. Review how GAIA currently limits requests and confirm the intended behavior for parallel evaluations; done means requests stay under the stated limit without delaying evaluations unnecessarily.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100