OpenHands / OpenHands/software-agent-sdk

[Harness Watch] Generate deterministic paired comparison reports

Open
#4,642 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement evaluation P0 performance
Dominant language
Python
Stars
1.1k
Forks
539
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Parent epic: #4627

Goal

Compare the four existing evaluation result archives deterministically. An LLM must not calculate metrics or decide whether a threshold was crossed.

P0

  • Validate matching instances, model/route, refs, budgets, harness identity/version, and complete telemetry.
  • Join cells by instance and report missing or invalid cells explicitly.
  • Compare OpenHands separately with OpenCode, Pi, and Hermes.
  • Report resolution difference and wins/losses/ties, with an exact paired binary test.
  • Report paired spend, proxy requests, input/cache/output tokens, and wall time using mean, median, P90, and one reproducible paired-bootstrap interval.
  • Report attempts, timeouts, infrastructure failures, telemetry coverage, and complete-pair count.
  • Emit versioned JSON and concise Markdown with the largest outcome/resource differences and links to trajectories and telemetry.
  • Test ties, missing cells, zero costs or successes, heavy tails, and bootstrap reproducibility.

Operational alarms use practical thresholds plus a confirmation run. A non-significant result is not proof of parity.

P1

Add multi-model and repeated-run models, cost-effectiveness views, dashboard/index publication, partial-quality signals, and historical analysis.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with parent epic #4627 and inspect the four existing evaluation result archives and their harness metadata. Define the deterministic validation, pairing, comparison, bootstrap, and JSON/Markdown reporting requirements in the P0 list. Done means reproducible reports explicitly cover invalid or missing cells, ties, resource metrics, failures, telemetry, and the named edge cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, analytics, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.