vllm-project / vllm-project/aibrix
Metric names from vLLM/SGLang may change across versions
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
We recently enhanced metric fetching robustness by updating `EngineMetricsNameMapping` from `map[string]string` to `map[string][]string`, allowing multiple candidate metric names per engine type. This fallback mechanism helps accommodate naming changes in inference engines like **vLLM** and **SGLang** across different releases (e.g., `vllm:time_per_output_token_seconds` → `vllm:inter_token_latency_seconds`). The improved logic is now implemented in `pkg/metrics/engine_fetcher.go` and `pkg/cache/cache_metrics.go`, and all metric definitions in `pkg/metrics/metrics.go` have been updated accordingly, including comments flagging deprecated vLLM metrics.
While this approach effectively handles *current* discrepancies, it raises a broader maintainability question:
> **How should we manage metric name in external inference engines?** 🤔
Unlike stable internal metrics, third-party engines may continue evolving their Prometheus metric names without backward compatibility guarantees.
Contributor guide
Assessment
This issue has not been assessed yet.