feat: add OpenTelemetry traces and metrics

Summary

Add opt-in OpenTelemetry (OTel) traces and metrics to vLLM Agentic API, with correlated local logs and OTLP export to a configurable Collector or compatible backend.

The gateway currently initializes a tracing log subscriber and emits scattered events, including request routing decisions. It lacks a consistent request trace and operational metrics for understanding whether failures or latency originate in inference, tool execution, state management, or client delivery.

The first milestone should make it possible to follow a request through the gateway, identify where time was spent, and distinguish successful execution from failures and disconnects—even after streaming HTTP headers have been sent.

Proposed implementation phases

1. OTel foundation

• Retain Rust tracing instrumentation and bridge spans through tracing-opentelemetry.
• Add opt-in OTLP trace and metric export with service identity, documented standard OTEL_* configuration, and configurable trace sampling.
• Keep SDK/provider initialization and exporter lifecycle in the server. Core instrumentation must work with providers supplied by embedding applications and must not initialize a global subscriber.
• Preserve local logs and include trace/span correlation where a request context exists. Keep local log filtering independent of trace sampling.
• Bound export buffering and record size, define overflow behavior, and ensure exporter outages do not fail requests or stall client delivery.
• Flush/shut down providers after request draining, with a bounded deadline.
• Verify dependency compatibility with the repository's Rust 1.85 minimum.

2. Request and execution traces

• Extract incoming W3C trace context and inject the active context into upstream inference requests.
• Cover both pass-through and executor paths across HTTP and WebSocket transports, including Responses and Messages execution where applicable.
• Add child spans for rehydration, inference rounds, gateway-executed built-in tools, compaction, and persistence.
• Preserve parentage across asynchronous tasks and streamed body polling.
• Keep streaming request spans alive through body completion or disconnect; represent execution and delivery outcomes separately when persistence or other work continues after a disconnect.
• Give each WebSocket request its own execution span rather than treating an entire connection as one request.
• Record typed failure categories and terminal outcomes; HTTP 200 alone must not imply execution success.

3. Metrics and operational validation

• Request counts, active requests, durations, and execution/delivery outcomes.
• Stage durations, inference-round counts, and token usage when upstream usage is available.
• Explicit timing definitions for first upstream data, first client-visible semantic event, and first text output where observable. Do not label all of these as time to first token.
• Client delivery waits, disconnects, and cancellation at the boundaries that own them.
• Ensure request metrics remain useful independently of trace sampling.
• Supply a backend-neutral local Collector example and deployment/configuration documentation.

Design constraints

• Do not capture prompts, model outputs, tool arguments, credentials, or arbitrary headers by default. Review existing events before exporting them as span events.
• Use bounded metric dimensions. Request IDs, conversation IDs, arbitrary tool names, and raw URLs must not become unbounded metric labels.
• Do not infer success or token usage when the upstream does not expose that information.
• Follow existing streaming ownership boundaries: inference transport owns HTTP/SSE framing, normalization and ingestion own semantic processing, the relay owns ordered client delivery, and the engine owns inference rounds, the tool loop, and persistence.
• Instrument the existing path; do not introduce additional parsers, assemblers, lifecycle validators, or delivery paths.
• Use standard semantic conventions where applicable; document project-specific attributes and metric names.

Acceptance criteria / test plan

• With telemetry disabled, the gateway serves requests without contacting an exporter.
• A Collector receives traces and metrics with configured service identity.
• Incoming trace context, gateway spans, and outbound inference context form the expected trace; absent or invalid context is handled safely.
• Multi-round requests show inference, tools, compaction, and state operations under the correct execution span.
• Streaming spans remain open beyond header emission and close correctly on completion, upstream failure, cancellation, or disconnect.
• Multiple requests on one WebSocket connection have distinct execution spans and correct metrics.
• Counters and active-request measurements finalize exactly once on normal and error paths.
• Metrics are still recorded when a trace is not sampled; unavailable usage is not fabricated.
• Tests verify that sensitive payloads and unbounded identifiers are excluded from exported attributes and metric dimensions.
• An unavailable/slow exporter and saturated export queue do not break request execution; shutdown finishes within its deadline.
• Representative streaming and non-streaming measurements document latency and memory overhead with telemetry disabled and enabled.
• Focused tests, formatting, Clippy, and repository pre-commit checks pass for the implementation.

Deferred / decisions to resolve

Native OTLP log export is a follow-up; correlated local logs are part of the first milestone. Backend-specific dashboards and alert rules are outside the initial implementation.

Resolve the initial OTLP transport(s), supported environment-variable set and precedence, sampling defaults, histogram boundaries, and any existing deployment Collector requirements during design. Use a backend-neutral Collector example unless deployment requirements dictate otherwise.

References

• Rust OpenTelemetry
• tracing-opentelemetry bridge
• OTLP exporter
• Streaming boundary work: #241, #243, #244, #245