Milestone 6: validate migrated harnesses against SWE-bench Pro
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
Work:
- Generate taskset configurations pairing SWE-bench Pro with each capability-compatible migrated harness.
- Validate sandbox connection, process cleanup, patch collection, observations, and result projection.
- Reject harnesses that cannot operate through the required sandbox capability before rollout.
- Add no new pairing logic to harnesses, SWE-bench Pro, rollout collection, or the single-agent processor.
Definition of done:
- Every compatible migrated harness runs SWE-bench Pro through configuration only.
- Every supported pairing has native rollout evidence.
- Every unsupported pairing produces a precise configuration-time capability error.
- A pairing failure that requires shared code is fixed once in the appropriate shared interface.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the taskset configuration flow and the migrated harnesses, then trace sandbox connection, process cleanup, patch collection, observations, and result projection. Done means every compatible pairing has native rollout evidence, while unsupported pairings fail at configuration time with a precise capability error and shared fixes are made once in the appropriate interface.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100