OpenHands / OpenHands/benchmarks
Port Commit0 benchmark to Harbor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 91
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Context
benchmarks/commit0 does not appear to have a corresponding Harbor registry dataset/adapter yet. To fully rely on Harbor for benchmark execution, Commit0 needs a Harbor adapter with parity against the current OpenHands benchmark implementation.
Proposed scope
- Create a Harbor adapter/dataset for Commit0 tasks.
- Preserve task setup, image/build behavior, verifier semantics, and current default dataset/split assumptions.
- Validate Oracle/reference behavior first, then compare an OpenHands/OpenHands SDK agent run against the existing harness.
- Document parity results and known exclusions.
Acceptance criteria
- Commit0 tasks can be run with
harbor runor an equivalent Harbor job config. - Parity results are recorded against the existing
benchmarks/commit0workflow. - The OpenHands wrapper can delegate Commit0 execution to Harbor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing benchmarks/commit0 workflow and its current default dataset, split, setup, build, and verifier behavior. Next, validate Oracle/reference behavior before comparing an OpenHands or OpenHands SDK agent run through Harbor. Done means a Harbor run configuration exists, parity results and exclusions are documented, and the OpenHands wrapper can delegate Commit0 execution to Harbor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100