OpenHands / OpenHands/benchmarks
Port SWEfficiency benchmark to Harbor
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 91
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Context
benchmarks/swefficiency does not appear to have a corresponding Harbor registry dataset/adapter yet. It has resource-sensitive Docker behavior and performance-oriented verification, so it should be ported deliberately with parity checks.
Proposed scope
- Create a Harbor adapter/dataset for SWEfficiency tasks.
- Preserve CPU/memory/resource assumptions and performance scoring semantics.
- Validate Oracle/reference behavior where available.
- Compare current OpenHands harness results against Harbor runs.
Acceptance criteria
- SWEfficiency tasks can be run through Harbor with appropriate resource controls.
- Performance/verifier parity is documented.
- The OpenHands wrapper can delegate SWEfficiency execution to Harbor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting benchmarks/swefficiency and the existing Harbor registry dataset/adapter patterns. Trace the OpenHands wrapper and compare its resource controls, verifier behavior, and performance scoring with Harbor runs. Done means SWEfficiency tasks run through Harbor, parity is documented, and the wrapper can delegate execution to Harbor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devtools, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100