OpenHands / OpenHands/benchmarks

Port HybridGym benchmarks to Harbor

Open
#724 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
90
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Context

The HybridGym benchmarks (hybridgym_funclocalize, hybridgym_funcgen, hybridgym_depsearch, and hybridgym_issuelocalize) do not appear to have corresponding Harbor registry datasets/adapters yet. These should be ported as a benchmark family while keeping per-task scoring behavior intact.

Proposed scope

  • Create Harbor adapters/tasks for each HybridGym variant.
  • Preserve prompts, environment setup, and evaluator semantics for each variant.
  • Validate parity separately for function localization, function generation, dependency search, and issue localization.
  • Document any shared adapter code and per-variant differences.

Acceptance criteria

  • All four HybridGym variants can be run through Harbor.
  • Per-variant parity results are recorded against the current OpenHands benchmark workflows.
  • OpenHands wrappers can delegate HybridGym execution to Harbor.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Harbor registry and adapter/task entry points for benchmark integrations, then inspect the current OpenHands benchmark workflows for the four HybridGym variants. Add coverage for function localization, function generation, dependency search, and issue localization while preserving each variant’s prompts, environment setup, and evaluator semantics. Done means all four run through Harbor, parity results are recorded, and OpenHands wrappers can delegate execution to Harbor.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.