OpenHands / OpenHands/benchmarks
Port HybridGym benchmarks to Harbor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Context
The HybridGym benchmarks (hybridgym_funclocalize, hybridgym_funcgen, hybridgym_depsearch, and hybridgym_issuelocalize) do not appear to have corresponding Harbor registry datasets/adapters yet. These should be ported as a benchmark family while keeping per-task scoring behavior intact.
Proposed scope
- Create Harbor adapters/tasks for each HybridGym variant.
- Preserve prompts, environment setup, and evaluator semantics for each variant.
- Validate parity separately for function localization, function generation, dependency search, and issue localization.
- Document any shared adapter code and per-variant differences.
Acceptance criteria
- All four HybridGym variants can be run through Harbor.
- Per-variant parity results are recorded against the current OpenHands benchmark workflows.
- OpenHands wrappers can delegate HybridGym execution to Harbor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Harbor registry and adapter/task entry points for benchmark integrations, then inspect the current OpenHands benchmark workflows for the four HybridGym variants. Add coverage for function localization, function generation, dependency search, and issue localization while preserving each variant’s prompts, environment setup, and evaluator semantics. Done means all four run through Harbor, parity results are recorded, and OpenHands wrappers can delegate execution to Harbor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100