OpenHands / OpenHands/benchmarks

create_docker_workspace never reuses locally-built images (tag mismatch with DockerDevWorkspace)

Open
#779 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
90
Avg merge
1d 6h
Merged PRs (30d)
1

Description

What's wrong

benchmarks/utils/image_utils.py:create_docker_workspace() pre-checks local_image_exists(agent_server_image) before falling back to building via the SDK's DockerDevWorkspace(base_image=..., target=...). But DockerDevWorkspace._build_image_from_base() (in openhands-workspace/openhands/workspace/docker/dev_workspace.py) builds via openhands.agent_server.docker.build.build(opts=BuildOptions(base_image=..., target=...)) without ever passing the agent_server_image tag. BuildOptions.image defaults to ghcr.io/openhands/agent-server (not this repo's EVAL_AGENT_SERVER_IMAGE = ghcr.io/openhands/eval-agent-server), and its tag is computed from base_tag (f"{short_sha}-{base_image_slug}") — a completely different naming scheme than whatever custom tag the calling benchmark computed.

Net effect: for every benchmark using create_docker_workspace with workspace_type=docker, if no pre-built image matches the expected tag, the fallback build tags its output under a name the next local_image_exists() check will never match — so local Docker runs always rebuild from scratch, even when an equivalent image was already built moments earlier in the same session.

Affected benchmarks

All current users of create_docker_workspace:

  • benchmarks/commit0/run_infer.py
  • benchmarks/gaia/run_infer.py
  • benchmarks/hybridgym_depsearch/run_infer.py
  • benchmarks/hybridgym_funcgen/run_infer.py
  • benchmarks/hybridgym_funclocalize/run_infer.py
  • benchmarks/hybridgym_issuelocalize/run_infer.py

Contrast with SWE-Bench

benchmarks/swebench/run_infer.py avoids this by using benchmarks/utils/build_utils.py:ensure_local_image() instead, which calls the benchmarks-side build_image() helper — that one builds and tags explicitly as target_image/custom_tag, matching what it just checked with local_image_exists().

Suggested fix

Either:

  1. Make create_docker_workspace's build fallback call ensure_local_image() (or build_image() directly) instead of DockerDevWorkspace, so the built tag matches the pre-checked one, or
  2. Pass image=/tag overrides into DockerDevWorkspace/BuildOptions so its output lands under the caller's expected agent_server_image tag.

Either way, this is local workspace_type=docker dev-loop ergonomics only — it doesn't affect CI-pushed remote images (workspace_type=remote), which use remote_image_exists() + pre-built GHCR tags directly.

Context

Found while reviewing #778 (dropping commit0's forked agent-layer Dockerfile) — commit0 was going to be brought in line with SWE-Bench's ensure_local_image pattern as part of that PR, but the fix was scoped out since it's a pre-existing, repo-wide gap shared by 6 benchmarks, not something specific to commit0's build pipeline.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in benchmarks/utils/image_utils.py at create_docker_workspace(), then compare its fallback with DockerDevWorkspace._build_image_from_base() in openhands-workspace/openhands/workspace/docker/dev_workspace.py. Review benchmarks/utils/build_utils.py:ensure_local_image() and the SWE-Bench caller for the expected tagging behavior. Done means a local Docker workspace reuses the agent_server_image tag after the fallback build instead of rebuilding in the six affected benchmarks.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.