OpenHands / OpenHands/benchmarks

feat: respect registry rate limits

Open
#172 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
90
Avg merge
1d 6h
Merged PRs (30d)
1

Description

We should add a rate limit checks to building images to avoid overwhelming docker.

A generic 429 Too Many Requests error, or a specific "Too Many Requests (HAP429)" error is generally referring to the abuse limit, described here: https://docs.docker.com/docker-hub/usage/#abuse-rate-limit
This limit applies to all requests to Hub properties including web pages, APIs, and image pulls. The limit is applied per-IP, and while the limit changes over time depending on load and other factors, it's in the order of thousands of requests per minute. The abuse limit applies to all users equally regardless of account level.

To avoid hitting this limit accidentally we should try to keep the api calls and image pulls to under 100 a minute.
Assuming we use 100 workers, this shouldn't delay evaluations. Even if we use 500 workers - unlikely due to LLM rate limits - it would only delay it by 4 minutes. Keeping it under 100 prevents accidentally reaching the 1000 limit when running multiple evaluations in parallel

This feature affects SWE-Bench, commit0, and SWT-bench and all feature benchmarks.

GAIA implements its own rate limit logic until (https://github.com/OpenHands/benchmarks/issues/171) is implemented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the image-building, registry API, and image-pull entry points used by SWE-Bench, commit0, SWT-bench, and feature benchmarks. Review how GAIA currently limits requests and confirm the intended behavior for parallel evaluations; done means requests stay under the stated limit without delaying evaluations unnecessarily.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.