vllm-project / vllm-project/aibrix
[RFC]: stormservice support vllm and sglang multiple-node deployment
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
Currently, we lack a unified multi-machine deployment solution. Raycluster currently supports multi-machine deployment using vllm, but does not support multi-machine deployment using sglang. The LeaderWorkerSet recommended by sglang lacks flexibility, so it is possible to use stormservice to support both types of inference.
Contributor guide
Research direction
Start by reviewing the existing RayCluster multi-machine deployment with vLLM and comparing it with the sglang LeaderWorkerSet approach described in the issue. Define how stormservice should support both inference engines across multiple machines, then validate the design against the flexibility and unified-deployment goals.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai-infra-agents, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100