vllm-project / vllm-project/agentic-api

feat: request, stage, and delivery metrics with a local Collector example (#279 phase 3)

Open
#340 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
284
Forks
74
Avg merge
1d 17h
Merged PRs (30d)
93

Description

Problem statement / motivation

Parent: #279 (phase 3). Depends on phases 1 #338 and 2 #339.

Operational metrics that are correct without trace sampling, with explicit timing definitions, bounded dimensions enforced by tests, a backend-neutral local Collector setup, and measured overhead with telemetry disabled vs enabled.

Proposed solution

WIP

Acceptance criteria (maps to #279 phase 3)
  • The Collector's debug exporter shows every instrument above with service.name and only allow-listed attribute keys/values.
  • Counters and agentic.execution.active finalize exactly once for completed, upstream-failure, disconnected, and shutdown-cancelled requests, over HTTP and WebSocket.
  • N requests on one WebSocket connection → N execution.count increments and one connections.active increment.
  • Metrics are recorded with OTEL_TRACES_SAMPLER_ARG=0; gen_ai.client.token.usage is absent when upstream omits usage — nothing fabricated.
  • The three first-* timings are distinct instruments; a cassette test proves first_upstream_data ≤ first_client_event ≤ first_text for a text response.
  • Cardinality test fails the build on any attribute outside the allow-list.
  • docker compose -f deploy/otel/docker-compose.yaml up + scripts/tests/otel-smoke.sh pass on a clean machine.
  • Overhead table (streaming and non-streaming, disabled vs enabled) in the docs
Non-goals

Dashboards/alert rules; exponential histograms (Collector-side); a Prometheus endpoint in the gateway (OTLP push only); per-tool-name / per-request-model / per-conversation / per-URL labels; OTLP log export.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the phase 3 acceptance criteria alongside parent #279 and dependencies #338 and #339. Run docker compose -f deploy/otel/docker-compose.yaml up and scripts/tests/otel-smoke.sh, then map the HTTP and WebSocket metric paths and the required tests. Done means every listed criterion passes, including bounded attributes, timing ordering, sampling behavior, smoke tests, and the documented overhead table.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, rust
Domain
backend-api-design, devops, documentation, observability, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.