OpenHands / OpenHands/benchmarks
Port OpenAgentSafety benchmark to Harbor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Context
benchmarks/openagentsafety does not appear to have a corresponding Harbor registry dataset/adapter yet. To remove benchmark-specific execution code from this repository, OpenAgentSafety needs a Harbor-native task representation and verifier.
Proposed scope
- Model OpenAgentSafety scenarios as Harbor tasks.
- Preserve NPC/workplace interaction setup and safety scoring semantics.
- Validate Oracle/reference behavior where available.
- Compare aggregate results with the current OpenHands benchmark harness.
Acceptance criteria
- OpenAgentSafety tasks can be run through Harbor.
- Verifier/scoring parity is documented.
- The OpenHands wrapper can delegate OpenAgentSafety execution to Harbor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting benchmarks/openagentsafety and the current OpenHands benchmark harness to understand the existing scenarios, wrapper, and scoring behavior. Then locate Harbor’s registry, task, and verifier integration points. Done means the scenarios run through Harbor, verifier and scoring parity is documented, aggregate results are compared, and the OpenHands wrapper delegates execution to Harbor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100