OpenHands / OpenHands/benchmarks

Port SWE-Bench Multimodal benchmark to Harbor

Open
#726 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
90
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Context

benchmarks/swebenchmultimodal does not appear to have a direct Harbor registry dataset/adapter yet. Harbor supports multimodal trajectories, but this benchmark needs a task adapter that preserves SWE-Bench Multimodal task setup, resolved-instance selection, and evaluation behavior.

Proposed scope

  • Create a Harbor adapter/dataset for SWE-Bench Multimodal.
  • Preserve the current resolved-instance selection behavior and multimodal assets.
  • Validate the verifier against the current swebenchmultimodal-infer/swebenchmultimodal-eval workflow.
  • Document any differences in image handling or multimodal output/trajectory handling.

Acceptance criteria

  • SWE-Bench Multimodal tasks can be run through Harbor.
  • Parity results are recorded against the existing OpenHands benchmark workflow.
  • The OpenHands wrapper can delegate SWE-Bench Multimodal execution to Harbor.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting benchmarks/swebenchmultimodal and the existing swebenchmultimodal-infer and swebenchmultimodal-eval workflow. Compare its task setup, resolved-instance selection, image handling, and trajectory behavior with Harbor and the OpenHands wrapper. Done means tasks run through Harbor, parity results are recorded, and the wrapper delegates execution successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.