vllm-project / vllm-project/aibrix

[RFC]: stormservice support vllm and sglang multiple-node deployment

Open
#1,647 3 comments 1 reaction 0 assignees View on GitHub
area/orchestration triage/accepted
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

Currently, we lack a unified multi-machine deployment solution. Raycluster currently supports multi-machine deployment using vllm, but does not support multi-machine deployment using sglang. The LeaderWorkerSet recommended by sglang lacks flexibility, so it is possible to use stormservice to support both types of inference.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing RayCluster multi-machine deployment with vLLM and comparing it with the sglang LeaderWorkerSet approach described in the issue. Define how stormservice should support both inference engines across multiple machines, then validate the design against the flexibility and unified-deployment goals.

Written by the indexing model from the issue text.

Assessment

Domain
ai-infra-agents, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.