OpenHands / OpenHands/software-agent-sdk
[Harness Watch] Capture comparable proxy telemetry
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 542
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 137
Description
Parent epic: #4627
Goal
Extend the existing per-attempt LiteLLM virtual-key lifecycle so native OpenHands and ACP harnesses have comparable provider-visible telemetry. This is an extension to the benchmark runner, not a new telemetry service.
P0
- Before deleting each virtual key, capture an allowlisted projection of every proxy request, including failed retries.
- Attribute telemetry to run, instance, harness, and attempt.
- Record request count, observed model/provider, spend, input/cache/output tokens, status, latency, and timestamps.
- Reconcile request-row spend with key spend. Mark missing, empty, partial, or unreconciled data as incomplete rather than zero.
- Reject unexpected model or routing.
- Update the existing efficiency summary to report proxy requests and stop presenting SDK cost entries as comparable iterations.
- Test pagination, delayed ledger writes, retries, zero-cost calls, reconciliation, and exclusion of prompts, responses, headers, and request bodies.
Acceptance
All four P0 harnesses produce reconciled telemetry on the shared smoke run, and failed attempts retain their cost and time.
P1
Add safe request-shape summaries, request-purpose classification, and a least-privilege telemetry endpoint if trajectory diagnosis needs them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parent epic #4627 and trace the existing benchmark runner, LiteLLM virtual-key lifecycle, and efficiency summary. Identify where keys are deleted and where summaries are assembled, then verify the shared smoke run across all four P0 harnesses; done means reconciled telemetry, retained failed-attempt cost and time, and tests covering pagination, retries, delayed writes, zero-cost calls, and data exclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100