[Harness Watch] Epic — automated paired ACP harness comparison

Goal

Automatically compare OpenHands, OpenCode, Kimi, Pi and Hermes on the same tasks and model. Measure outcome, provider-visible spend, requests, tokens, and time; confirm large differences before reporting them.

Approach

Reuse the evaluation system that already works:

• software-agent-sdk owns ACP harness support and pinned binaries.
• benchmarks owns the fixed instance panel and per-attempt proxy telemetry.
• evaluation owns one thin scheduled controller around the existing eval-job.yml and a deterministic paired report.
• Each harness runs as a separate existing evaluation job with the same resolved refs, model configuration, instances, and budgets.

P0 uses one shared model and one committed panel of roughly 30–50 instances. No historical backfill is required.

P0 issues

• #4638 — Capture comparable proxy telemetry
• #4639 — Add OpenCode through the existing ACP path
• #4860 — Wire Kimi into the eval path (closed — SDK merged in #4714; eval wiring in #4833)
• #4635 — Add Pi through the existing ACP path (closed — SDK merged in #4419; eval wiring in #4833)
• #4634 — Add Hermes through the existing ACP path
• #4637 — Schedule and confirm comparisons using the existing evaluation workflow
• #4642 — Generate the paired comparison report

OpenCode should produce the first vertical slice. Kimi and Pi are the cheapest to add — both are already merged SDK providers, so only the eval wiring remains; Hermes needs a small SDK change first (#4634).

Definition of done

A weekly and manual run can:

1. resolve the experiment configuration once;
2. dispatch the existing evaluation workflow for all four harnesses on the same panel;
3. reject cells with configuration drift, missing results, or incomplete telemetry;
4. publish an auditable pairwise OpenHands comparison;
5. rerun OpenHands and an affected comparator once before reporting a material difference.

P1

• #4641 — Read-only trajectory diagnosis and experiment proposal
• Multiple models, rotating panels, advanced sequential statistics, dashboards, index publication, and historical analysis

Guardrails

• A comparable inference unit is a LiteLLM proxy request, not an ACP turn or claimed agent iteration.
• Preserve failed attempts and intent-to-treat cost and time.
• Missing telemetry is incomplete, never zero.
• Compare OpenHands separately with each other harness; do not invent a global winner.
• Conclusions remain specific to the selected model and panel.

Start with the existing eval-job.yml workflow, then trace ACP harness support in software-agent-sdk and the fixed instance panel and proxy telemetry in benchmarks. Review how evaluation currently dispatches jobs before assessing the scheduled controller and deterministic paired report. Done means the listed P0 issues support comparable, auditable OpenHands comparisons with incomplete or drifting cells rejected.