vllm-project / vllm-project/aibrix
Multi-Node Inference ' Waiting for creating a placement group of specs for 310 seconds'
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🚀 Feature Description and Motivation
```
NING 04-01 02:45:50 ray_utils.py:320] The number of required GPUs exceeds the total number of available GPUs in the placement group.
INFO 04-01 02:46:00 ray_utils.py:214] Waiting for creating a placement group of specs for 10 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:46:20 ray_utils.py:214] Waiting for creating a placement group of specs for 30 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:47:00 ray_utils.py:214] Waiting for creating a placement group of specs for 70 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:48:20 ray_utils.py:214] Waiting for creating a placement group of specs for 150 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:51:00 ray_utils.py:214] Waiting for creating a placement group of specs for 310 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:56:20 ray_utils.py:214] Waiting for creating a placement group of specs for 630 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
```
### Use Case
I use the code`https://github.com/vllm-project/aibrix/blob/main/samples/distributed/fleet-two-node.yaml` run ray examble
### Proposed Solution
_No response_
Contributor guide
Research direction
Start with samples/distributed/fleet-two-node.yaml and the ray_utils.py locations shown in the logs, especially lines 214 and 320. Reproduce the example, inspect ray status, and verify that cluster IP addresses match VLLM_HOST_IP on each node; done means the placement group is created without indefinite waiting.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai-infra-agents, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100