OpenHands / OpenHands/benchmarks
Port SWE-Bench Multimodal benchmark to Harbor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Context
benchmarks/swebenchmultimodal does not appear to have a direct Harbor registry dataset/adapter yet. Harbor supports multimodal trajectories, but this benchmark needs a task adapter that preserves SWE-Bench Multimodal task setup, resolved-instance selection, and evaluation behavior.
Proposed scope
- Create a Harbor adapter/dataset for SWE-Bench Multimodal.
- Preserve the current resolved-instance selection behavior and multimodal assets.
- Validate the verifier against the current
swebenchmultimodal-infer/swebenchmultimodal-evalworkflow. - Document any differences in image handling or multimodal output/trajectory handling.
Acceptance criteria
- SWE-Bench Multimodal tasks can be run through Harbor.
- Parity results are recorded against the existing OpenHands benchmark workflow.
- The OpenHands wrapper can delegate SWE-Bench Multimodal execution to Harbor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting benchmarks/swebenchmultimodal and the existing swebenchmultimodal-infer and swebenchmultimodal-eval workflow. Compare its task setup, resolved-instance selection, image handling, and trajectory behavior with Harbor and the OpenHands wrapper. Done means tasks run through Harbor, parity results are recorded, and the wrapper delegates execution successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100