vllm-project / vllm-project/aibrix

stormservice deploy use mutiple node

Open
#1,604 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

I deploy stormservice with the podgroupsize: 2, it seem that the pod belong to the same podset donnot common inference the model.Is there demo for the case?
Image
```
INFO 09-25 00:16:49 [ray_utils.py:334] No current placement group found. Creating a new placement group.
WARNING 09-25 00:16:49 [ray_utils.py:341] The number of required GPUs exceeds the total number of available GPUs in the placement group.
INFO 09-25 00:16:59 [ray_utils.py:232] Waiting for creating a placement group of specs for 10 seconds. specs=[{'GPU': 1.0, 'node:xx': 0.001}, {'GPU': 1.0}]. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 09-25 00:17:19 [ray_utils.py:232] Waiting for creating a placement group of specs for 30 seconds. specs=[{'GPU': 1.0, 'node:xx': 0.001}, {'GPU': 1.0}]. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 09-25 00:17:59 [ray_utils.py:232] Waiting for creating a placement group of specs for 70 seconds. specs=[{'GPU': 1.0, 'node:xxx': 0.001}, {'GPU': 1.0}]. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
```

### Use Case

```
spec:
roles:
- name: prefill
replicas: 2
podGroupSize: 2
stateful: true
template:
metadata:
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
labels:
model.aibrix.ai/name: qwen3-8B-podset
model.aibrix.ai/port: "8000"
model.aibrix.ai/engine: vllm
spec:
containers:
- name: prefill
image: kvcache-container-image-hb2-cn-beijing.cr.volces.com/aibrix/vllm-openai:v0.9.2-cu128-nixl-v0.4.1-lmcache-0.3.1.post1
command: ["sh", "-c"]
args:
- |
python3 -m vllm.entrypoints.openai.api_server \
--host "0.0.0.0" \
--port "8000" \
--uvicorn-log-level warning \
--model /models/qwen/Qwen3-32B \
--gpu-memory-utilization 0.9 \
--tensor-parallel-size 1 \
--pipeline-parallel-size 2 \
--served-model-name qwen3-8B-podset \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
env:
- name: PYTHONHASHSEED
value: "1047"
- name: VLLM_SERVER_DEV_MODE
value: "1"
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
value: "0.0.0.0"
- name: VLLM_NIXL_SIDE_CHANNEL_PORT
value: "5558"
- name: VLLM_WORKER_MULTIPROC_METHOD
value: spawn
- name: VLLM_ENABLE_V1_MULTIPROCESSING
value: "0"
- name: GLOO_SOCKET_IFNAME
value: eth0
- name: NCCL_SOCKET_IFNAME
value: eth0
- name: NCCL_IB_DISABLE
value: "0"
- name: NCCL_IB_GID_INDEX
value: "7"
- name: NCCL_DEBUG
value: "INFO"
- name: UCX_TLS
value: ^gga
volumeMounts:
- mountPath: /dev/shm
name: shared-mem
resources:
limits:
nvidia.com/gpu: 1
securityContext:
capabilities:
add:
- IPC_LOCK
schedulerName: volcano
- name: decode
replicas: 1
podGroupSize: 2
stateful: true
template:
metadata:
annotations:
k8s.volcengine.com/pod-networks: |
[
{
"cniConf":{
"name":"rdma"
}
}
]
labels:
model.aibrix.ai/name: qwen3-8B-podset
model.aibrix.ai/port: "8000"
model.aibrix.ai/engine: vllm
spec:
containers:
- name: decode
image: kvcache-container-image-hb2-cn-beijing.cr.volces.com/aibrix/vllm-openai:v0.9.2-cu128-nixl-v0.4.1-lmcache-0.3.1.post1
command: ["sh", "-c"]
args:
- |
python3 -m vllm.entrypoints.openai.api_server \
--host "0.0.0.0" \
--port "8000" \
--uvicorn-log-level warning \
--model models/qwen/Qwen3-32B \
--gpu-memory-utilization 0.9 \
--tensor-parallel-size 1 \
--pipeline-parallel-size 2 \
--served-model-name qwen3-8B-podset \
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
env:
- name: PYTHONHASHSEED
value: "1047"
- name: VLLM_SERVER_DEV_MODE
value: "1"
- name: VLLM_NIXL_SIDE_CHANNEL_HOST
value: "0.0.0.0"
- name: VLLM_NIXL_SIDE_CHANNEL_PORT
value: "5558"
- name: VLLM_WORKER_MULTIPROC_METHOD
value: spawn
- name: VLLM_ENABLE_V1_MULTIPROCESSING
value: "0"
- name: GLOO_SOCKET_IFNAME
value: eth0
- name: NCCL_SOCKET_IFNAME
value: eth0
- name: NCCL_IB_DISABLE
value: "0"
- name: NCCL_IB_GID_INDEX
value: "7"
- name: NCCL_DEBUG
value: "INFO"
- name: UCX_TLS
value: ^gga
```
Otherwise prefill-b7d7f85cc-0-0 and prefill-b7d7f85cc-0-1 start the model separately as a pod

### Proposed Solution
@Jeffwan
_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Kubernetes role specification and the ray_utils.py log locations at lines 232, 334, and 341. Reproduce the multi-node deployment with podGroupSize: 2, then trace how the prefill pods are placed and started. Done should define and demonstrate the expected shared inference behavior for pods in one podset.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
ai-infra-agents, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.