OpenHands / OpenHands/benchmarks
orchestrate_eval.py crashes when benchmarks_branch is a commit SHA instead of a branch name
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Bug
orchestrate_eval.py's create_temporary_branch() function fails with a 404 error when BENCHMARKS_REF is a commit SHA instead of a branch name.
Root Cause
The function resolves the base ref via the GitHub Refs API:
GET https://api.github.com/repos/{repo}/git/refs/heads/{BENCHMARKS_REF}
This endpoint expects a branch name (e.g., main), not a raw commit SHA. When a 40-character SHA is passed, no matching ref exists and the API returns 404.
Reproduction
Trigger an eval with benchmarks_branch set to a commit SHA:
inputs[benchmarks_branch]=641119486f8009193e396df71cfcd16817eb0904
The job fails immediately:
[2026-03-27 01:47:34 UTC] Using base benchmarks ref: 641119486f8009193e396df71cfcd16817eb0904
[2026-03-27 01:47:35 UTC] ERROR: Failed to get base branch SHA (status 404): {"message":"Not Found",...}
Workaround
Create a branch from the SHA first, then pass the branch name instead.
Suggested Fix
In create_temporary_branch(), detect when BENCHMARKS_REF is a SHA (40-char hex string) and use it directly as the base_sha for branch creation, skipping the git/refs/heads/ lookup.
import re
if re.fullmatch(r'[0-9a-f]{40}', benchmarks_ref):
base_sha = benchmarks_ref
else:
# existing ref lookup logic
...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in orchestrate_eval.py at create_temporary_branch() and inspect the existing GitHub refs lookup. Reproduce the evaluation with the provided 40-character benchmarks_branch value, then verify that SHA-based inputs create the temporary branch without the 404 while branch-name inputs retain their existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100