vllm-project / vllm-project/production-stack

feature: Design a Dedicated Metrics Module for Engine/Request Metrics

Open
#494 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Python
Stars
2.6k
Forks
503
Avg merge
4d 17h
Merged PRs (30d)
8

Description

Describe the feature

Currently, metrics related to engine/request in the production-stack are collected within the router——a responsibility that should ideally belong to a dedicated observability layer, This architecture leads to:

  1. Violation of separation of concerns: Router's core function is routing, not metrics management
  2. Duplicate collection overhead: Both HPA and observability systems need to scrape metrics from router, creating unnecessary 2x load
  3. Tight coupling: Metrics format changes require router modifications

Implementing centralized metrics storage would:

  1. Enable single-point collection for all consumers (cluster obser #474, router scheduling, HPA #238)
  2. Reduce router CPU/memory pressure by offloading metrics work
  3. Standardize metrics access across subsystems
Why do you need this feature?

Design a dedicated metrics module/database (similar to etcd/redis) that would:

  1. Collect and store all engine/request metrics centrally
  2. Provide a unified interface for metrics querying
  3. Support both real-time and historical metrics analysis
Additional context

Do you think a separate metrics module is necessary? I'm still learning about system work, so I'd be grateful for your corrections or suggestions!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating the router's current engine/request metric collection and reading related issues #474 and #238; the issue does not define a concrete implementation or an unambiguous definition of done.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.