ray ports exhausted when using async vllm on some clusters
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
With the ray port range below, we can run well w/ both vllm sync or async engine on our cluster, but will sometimes fail w/ vllm async engine in some clusters.
```
MIN_WORKER_PORT=${MIN_WORKER_PORT:-54001}
MAX_WORKER_PORT=${MAX_WORKER_PORT:-54257}
```
We have a workaround https://github.com/NVIDIA-NeMo/RL/pull/950 that expand the range from 54001 ~ 54257 to 54001 ~ 54513 for now.
**Some investigation**
From one run, before starting `uv run python examples/run_grpo_math.py`, all the ports in 54001 to 54257 are free, and after async vllm init, the ports are almost exhausted. then after a while ray complains no port available.
**Error info**
`core_worker.cc:513: Failed to register worker to Raylet: Invalid: Invalid: No available ports. Please specify a wider port range using --min-worker-port and --max-worker-port. `
Contributor guide
Assessment
This issue has not been assessed yet.