OpenHands / OpenHands/software-agent-sdk
[Harness Watch] Generate deterministic paired comparison reports
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 539
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Parent epic: #4627
Goal
Compare the four existing evaluation result archives deterministically. An LLM must not calculate metrics or decide whether a threshold was crossed.
P0
- Validate matching instances, model/route, refs, budgets, harness identity/version, and complete telemetry.
- Join cells by instance and report missing or invalid cells explicitly.
- Compare OpenHands separately with OpenCode, Pi, and Hermes.
- Report resolution difference and wins/losses/ties, with an exact paired binary test.
- Report paired spend, proxy requests, input/cache/output tokens, and wall time using mean, median, P90, and one reproducible paired-bootstrap interval.
- Report attempts, timeouts, infrastructure failures, telemetry coverage, and complete-pair count.
- Emit versioned JSON and concise Markdown with the largest outcome/resource differences and links to trajectories and telemetry.
- Test ties, missing cells, zero costs or successes, heavy tails, and bootstrap reproducibility.
Operational alarms use practical thresholds plus a confirmation run. A non-significant result is not proof of parity.
P1
Add multi-model and repeated-run models, cost-effectiveness views, dashboard/index publication, partial-quality signals, and historical analysis.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parent epic #4627 and inspect the four existing evaluation result archives and their harness metadata. Define the deterministic validation, pairing, comparison, bootstrap, and JSON/Markdown reporting requirements in the P0 list. Done means reproducible reports explicitly cover invalid or missing cells, ties, resource metrics, failures, telemetry, and the named edge cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, analytics, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100