vllm-project / vllm-project/aibrix

Metric names from vLLM/SGLang may change across versions

Open
#1,815 1 comment 0 reactions 0 assignees View on GitHub
area/gateway
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

We recently enhanced metric fetching robustness by updating `EngineMetricsNameMapping` from `map[string]string` to `map[string][]string`, allowing multiple candidate metric names per engine type. This fallback mechanism helps accommodate naming changes in inference engines like **vLLM** and **SGLang** across different releases (e.g., `vllm:time_per_output_token_seconds` → `vllm:inter_token_latency_seconds`). The improved logic is now implemented in `pkg/metrics/engine_fetcher.go` and `pkg/cache/cache_metrics.go`, and all metric definitions in `pkg/metrics/metrics.go` have been updated accordingly, including comments flagging deprecated vLLM metrics.

While this approach effectively handles *current* discrepancies, it raises a broader maintainability question:

> **How should we manage metric name in external inference engines?** 🤔

Unlike stable internal metrics, third-party engines may continue evolving their Prometheus metric names without backward compatibility guarantees.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.