vllm-project / vllm-project/production-stack
Discussion - QPS routing when there are multiple router replicas
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 503
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 8
Description
When multiple router replicas are deployed, the QPS routing on a router replica only relies on its in-memory Request Stats.
This means that the routing logic doesn't consider the overall QPS on the serving URL but only the QPS routed via this replica.
This might cause the overload the URL when it has more requests coming through other router replicas.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or specific entry points. Start by locating the router's in-memory Request Stats and the QPS routing path, then examine how traffic is represented across replicas. Done requires an agreed design for accounting for overall serving-URL QPS and a clear validation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100