vllm-project / vllm-project/aibrix

Multi-Node Inference ' Waiting for creating a placement group of specs for 310 seconds'

Open
#927 3 comments 0 reactions 0 assignees View on GitHub
area/distributed
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
104

Description

### 🚀 Feature Description and Motivation

```
NING 04-01 02:45:50 ray_utils.py:320] The number of required GPUs exceeds the total number of available GPUs in the placement group.
INFO 04-01 02:46:00 ray_utils.py:214] Waiting for creating a placement group of specs for 10 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:46:20 ray_utils.py:214] Waiting for creating a placement group of specs for 30 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:47:00 ray_utils.py:214] Waiting for creating a placement group of specs for 70 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:48:20 ray_utils.py:214] Waiting for creating a placement group of specs for 150 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:51:00 ray_utils.py:214] Waiting for creating a placement group of specs for 310 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
INFO 04-01 02:56:20 ray_utils.py:214] Waiting for creating a placement group of specs for 630 seconds. specs=[{'node:10.163.194.101': 0.001, 'GPU': 1.0}, {'GPU': 1.0}]. Check `ray status` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.
```

### Use Case

I use the code`https://github.com/vllm-project/aibrix/blob/main/samples/distributed/fleet-two-node.yaml` run ray examble

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with samples/distributed/fleet-two-node.yaml and the ray_utils.py locations shown in the logs, especially lines 214 and 320. Reproduce the example, inspect ray status, and verify that cluster IP addresses match VLLM_HOST_IP on each node; done means the placement group is created without indefinite waiting.

Written by the indexing model from the issue text.

Assessment

Domain
ai-infra-agents, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.