NVIDIA-NeMo / NVIDIA-NeMo/RL

ray ports exhausted when using async vllm on some clusters

Open
#951 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

With the ray port range below, we can run well w/ both vllm sync or async engine on our cluster, but will sometimes fail w/ vllm async engine in some clusters.
```
MIN_WORKER_PORT=${MIN_WORKER_PORT:-54001}
MAX_WORKER_PORT=${MAX_WORKER_PORT:-54257}
```

We have a workaround https://github.com/NVIDIA-NeMo/RL/pull/950 that expand the range from 54001 ~ 54257 to 54001 ~ 54513 for now.

**Some investigation**
From one run, before starting `uv run python examples/run_grpo_math.py`, all the ports in 54001 to 54257 are free, and after async vllm init, the ports are almost exhausted. then after a while ray complains no port available.

**Error info**
`core_worker.cc:513: Failed to register worker to Raylet: Invalid: Invalid: No available ports. Please specify a wider port range using --min-worker-port and --max-worker-port. `

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.