Integrate SWE-Bench Verified
Open
Nobody has claimed this yet.
community-request
env-infra
needs-design
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 318
Description
Background
SWE-Bench Verified is a popular benchmark for evaluating agents on real-world software engineering tasks.
What is SWE-Bench Verified
SWE-Bench Verified contains GitHub issues from open-source projects where agents must understand the codebase, write fixes, and pass test suites. The "Verified" subset has high-quality, human-validated examples. https://www.swebench.com/
Acceptance Criteria
- Implement SWE-Bench Verified in NeMo Gym
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named in the issue. Start by reviewing NeMo Gym's existing environment and benchmark integrations, then read the SWE-Bench Verified documentation linked in the issue. Done means SWE-Bench Verified is implemented in NeMo Gym and its evaluation path is usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100