[Feature]: Add Metrics to Support k8s Inference Gateway Scheduling in trtllm-serve

🚀 The feature, motivation and pitch

Feature

Add TotalQueuedRequests and TotalRunningRequests Prometheus gauge metrics to trtllm-serve.

Currently, trtllm-serve (as of 1.3.0rc7) exposes Prometheus metrics via /prometheus/metrics, but is missing two gauge metrics required by the Kubernetes Inference Gateway for load-aware routing:

• trtllm_num_requests_waiting (Gauge) — Current number of requests in the queue
• trtllm_num_requests_running (Gauge) — Current number of requests actively being served

Motivation

The Kubernetes Gateway API Inference Extension (aka Inference Gateway) scrapes Prometheus metrics from model servers to make routing
decisions. Its model server protocol requires three metrics:
TotalQueuedRequests, TotalRunningRequests, and KVCacheUtilization.

trtllm-serve currently only exposes trtllm_kv_cache_utilization — the other two are missing. trtllm_request_queue_time_seconds exists but is a histogram (latency), not a gauge (count), so it cannot be
used as a substitute.

The Triton TRT-LLM backend already exposes all three required metrics. Other model servers (vLLM, sglang) also expose equivalent gauges. trtllm-serve is the only supported frontend missing them.

A comparison of trtllm-serve vs Triton metric names and the full gap analysis can be found in
kubernetes-sigs/gateway-api-inference-extension#2596.

Alternatives

No response

Additional context

The implementation should be straightforward. The MetricsCollector class in tensorrt_llm/metrics/collector.py already handles iteration stats via log_iteration_stats(). The engine's get_stats()
likely already provides request counts in its iteration stats dict — they just need to be:

1. Added as Gauge metrics in MetricsCollector.__init__()
2. Logged in MetricsCollector.log_iteration_stats() from the appropriate fields in the stats dict

For reference, other model servers expose these as:

• vLLM: vllm:num_requests_waiting, vllm:num_requests_running
• sglang: sglang:num_queue_reqs, sglang:num_running_reqs
• Triton TRT-LLM: nv_trt_llm_request_metrics{request_type="waiting"}, nv_trt_llm_request_metrics{request_type="scheduled"}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.