OpenHands / OpenHands/software-agent-sdk

[Harness Watch] Capture comparable proxy telemetry

Open
#4,638 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement evaluation P0 performance
Dominant language
Python
Stars
1.1k
Forks
542
Avg merge
1d 19h
Merged PRs (30d)
137

Description

Parent epic: #4627

Goal

Extend the existing per-attempt LiteLLM virtual-key lifecycle so native OpenHands and ACP harnesses have comparable provider-visible telemetry. This is an extension to the benchmark runner, not a new telemetry service.

P0

  • Before deleting each virtual key, capture an allowlisted projection of every proxy request, including failed retries.
  • Attribute telemetry to run, instance, harness, and attempt.
  • Record request count, observed model/provider, spend, input/cache/output tokens, status, latency, and timestamps.
  • Reconcile request-row spend with key spend. Mark missing, empty, partial, or unreconciled data as incomplete rather than zero.
  • Reject unexpected model or routing.
  • Update the existing efficiency summary to report proxy requests and stop presenting SDK cost entries as comparable iterations.
  • Test pagination, delayed ledger writes, retries, zero-cost calls, reconciliation, and exclusion of prompts, responses, headers, and request bodies.

Acceptance

All four P0 harnesses produce reconciled telemetry on the shared smoke run, and failed attempts retain their cost and time.

P1

Add safe request-shape summaries, request-purpose classification, and a least-privilege telemetry endpoint if trajectory diagnosis needs them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with parent epic #4627 and trace the existing benchmark runner, LiteLLM virtual-key lifecycle, and efficiency summary. Identify where keys are deleted and where summaries are assembled, then verify the shared smoke run across all four P0 harnesses; done means reconciled telemetry, retained failed-attempt cost and time, and tests covering pagination, retries, delayed writes, zero-cost calls, and data exclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.