vllm-project / vllm-project/agentic-api
feat: request, stage, and delivery metrics with a local Collector example (#279 phase 3)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 284
- Forks
- 74
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 93
Description
Problem statement / motivation
Parent: #279 (phase 3). Depends on phases 1 #338 and 2 #339.
Operational metrics that are correct without trace sampling, with explicit timing definitions, bounded dimensions enforced by tests, a backend-neutral local Collector setup, and measured overhead with telemetry disabled vs enabled.
Proposed solution
WIP
Acceptance criteria (maps to #279 phase 3)
- The Collector's
debugexporter shows every instrument above withservice.nameand only allow-listed attribute keys/values. - Counters and
agentic.execution.activefinalize exactly once for completed, upstream-failure, disconnected, and shutdown-cancelled requests, over HTTP and WebSocket. - N requests on one WebSocket connection → N
execution.countincrements and oneconnections.activeincrement. - Metrics are recorded with
OTEL_TRACES_SAMPLER_ARG=0;gen_ai.client.token.usageis absent when upstream omitsusage— nothing fabricated. - The three first-* timings are distinct instruments; a cassette test proves
first_upstream_data ≤ first_client_event ≤ first_textfor a text response. - Cardinality test fails the build on any attribute outside the allow-list.
-
docker compose -f deploy/otel/docker-compose.yaml up+scripts/tests/otel-smoke.shpass on a clean machine. - Overhead table (streaming and non-streaming, disabled vs enabled) in the docs
Non-goals
Dashboards/alert rules; exponential histograms (Collector-side); a Prometheus endpoint in the gateway (OTLP push only); per-tool-name / per-request-model / per-conversation / per-URL labels; OTLP log export.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the phase 3 acceptance criteria alongside parent #279 and dependencies #338 and #339. Run docker compose -f deploy/otel/docker-compose.yaml up and scripts/tests/otel-smoke.sh, then map the HTTP and WebSocket metric paths and the required tests. Done means every listed criterion passes, including bounded attributes, timing ordering, sampling behavior, smoke tests, and the documented overhead table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, rust
- Domain
- backend-api-design, devops, documentation, observability, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100