OpenHands / OpenHands/benchmarks

orchestrate_eval.py crashes when benchmarks_branch is a commit SHA instead of a branch name

Open Beginner friendly
#581 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
90
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Bug

orchestrate_eval.py's create_temporary_branch() function fails with a 404 error when BENCHMARKS_REF is a commit SHA instead of a branch name.

Root Cause

The function resolves the base ref via the GitHub Refs API:

GET https://api.github.com/repos/{repo}/git/refs/heads/{BENCHMARKS_REF}

This endpoint expects a branch name (e.g., main), not a raw commit SHA. When a 40-character SHA is passed, no matching ref exists and the API returns 404.

Reproduction

Trigger an eval with benchmarks_branch set to a commit SHA:

inputs[benchmarks_branch]=641119486f8009193e396df71cfcd16817eb0904

The job fails immediately:

[2026-03-27 01:47:34 UTC] Using base benchmarks ref: 641119486f8009193e396df71cfcd16817eb0904
[2026-03-27 01:47:35 UTC] ERROR: Failed to get base branch SHA (status 404): {"message":"Not Found",...}

Workaround

Create a branch from the SHA first, then pass the branch name instead.

Suggested Fix

In create_temporary_branch(), detect when BENCHMARKS_REF is a SHA (40-char hex string) and use it directly as the base_sha for branch creation, skipping the git/refs/heads/ lookup.

import re

if re.fullmatch(r'[0-9a-f]{40}', benchmarks_ref):
    base_sha = benchmarks_ref
else:
    # existing ref lookup logic
    ...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in orchestrate_eval.py at create_temporary_branch() and inspect the existing GitHub refs lookup. Reproduce the evaluation with the provided 40-character benchmarks_branch value, then verify that SHA-based inputs create the temporary branch without the 404 while branch-name inputs retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.