NVIDIA-NeMo / NVIDIA-NeMo/RL

GRPO Quick Start guide missing prerequisites, env vars, and troubleshooting for single-node H20 setup

Open
#2,043 4 comments 0 reactions 1 assignee Claimed by @ZhiyuLi-Nvidia View on GitHub
bug community-request
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

Following the [GRPO Quick Start guide](https://github.com/NVIDIA-NeMo/RL/blob/main/docs/guides/grpo.md#quickstart-launch-a-grpo-run) on an H20 single-node (8 GPUs) setup, I encountered 9 separate dependency/environment errors before successfully running training. The guide does not mention required `uv sync` extras, necessary environment variables, or common troubleshooting steps.

**Environment:**
- GPU: NVIDIA H20 (8x, single node)
- Container: Built from `docker/Dockerfile` on `main` branch
- NeMo RL: main branch (latest)
- Build command: `docker buildx build -f docker/Dockerfile -t nemo-rl:latest .`

**Steps/Code to reproduce bug**

1. Build the Docker image:
```sh
docker buildx build -f docker/Dockerfile -t nemo-rl:latest .
```

2. Start the container:
```sh
docker run -it --gpus all nemo-rl:latest bash
```

3. Run GRPO as documented:
```sh
uv run examples/run_grpo.py --config examples/configs/grpo_math_1B.yaml cluster.gpus_per_node=8
```

4. Encounter the following errors sequentially:

| # | Error | Root Cause |
|---|-------|-----------|
| 1 | `ImportError: cannot import name 'OmegaConf'` | venv not synced |
| 2 | `ModuleNotFoundError: No module named 'ray._private.node'` | Ray installation incomplete |
| 3 | `ModuleNotFoundError: No module named 'nemo_automodel'` | `automodel` extra not mentioned in quickstart |
| 4 | `AttributeError: module 'uvloop' has no attribute 'install'` | uvloop version incompatible with Ray 2.49.2 |
| 5 | `KeyError: 'Version'` (email-validator) | Package metadata corruption |
| 6 | `ModuleNotFoundError: No module named 'vllm.logger'` | vLLM installation incomplete |
| 7 | `AttributeError: module 'tabulate' has no attribute 'tabulate'` | tabulate version too old |
| 8 | `OSError: libcudnn.so.9: cannot open shared object file` | cuDNN not visible in Ray worker venvs (`/opt/ray_venvs/`) |
| 9 | `RuntimeError: DeepGEMM backend is not available` | DeepGEMM not compiled for H20 |

**Expected behavior**

The Quick Start guide should either:
1. Work out of the box after building the Docker image, OR
2. Clearly document the required setup steps (extras, env vars) before the launch command

Specifically, the guide should mention:
- Running `uv sync --extra automodel --extra vllm` before launching
- Setting `VLLM_USE_DEEP_GEMM=0` for GPUs where DeepGEMM is not available (e.g., H20)
- Setting `LD_LIBRARY_PATH` so Ray workers can find cuDNN
- Common error symptoms and their fixes

**Additional context**

I have documentation fixes ready for 3 files and plan to submit a PR:
- `docs/guides/grpo.md`: Add Prerequisites section and Troubleshooting table
- `docs/docker.md`: Add "Verifying the Build" section
- `docs/about/installation.md`: Add note about cuDNN in Ray worker venvs

For deeper fixes (dependency pinning in `pyproject.toml`, cuDNN handling in `nemo_rl/utils/venv.py`, Dockerfile validation), I plan separate follow-up PRs.

Happy to adjust the direction based on maintainer feedback before submitting.

Full Traceback: Issue #1 — ImportError: cannot import name 'OmegaConf'

```
uv run python examples/run_grpo.py
Traceback (most recent call last):
File "/opt/nemo-rl/examples/run_grpo.py", line 19, in
from omegaconf import OmegaConf
ImportError: cannot import name 'OmegaConf' from 'omegaconf' (unknown location)
```

Full Traceback: Issue #2 — ModuleNotFoundError: No module named 'ray._private.node'

```
Traceback (most recent call last): [repeated 8x across cluster]
(raylet) File "/opt/nemo_rl_venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py", line 9, in
(raylet) import ray._private.node
(raylet) ModuleNotFoundError: No module named 'ray._private.node' [repeated 8x across cluster]
```

Full Traceback: Issue #3 — ModuleNotFoundError: No module named 'nemo_automodel'

```
File "", line 935, in _load_unlocked
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "/opt/nemo-rl/nemo_rl/models/policy/workers/dtensor_policy_worker_v2.py", line 23, in
from nemo_automodel.components._peft.lora import LinearLoRA
ModuleNotFoundError: No module named 'nemo_automodel'
```

Full Traceback: Issue #4 — AttributeError: module 'uvloop' has no attribute 'install'

```
Using sequential worker initialization (colocated mode)
Initializing vllm_policy workers: 0%| | 0/8 [00:00
(raylet) try_install_uvloop()
(raylet) File "/opt/nemo_rl_venv/lib/python3.12/site-packages/ray/_private/async_compat.py", line 26, in try_install_uvloop
(raylet) uvloop.install()
(raylet) ^^^^^^^^^^^^^^
(raylet) AttributeError: module 'uvloop' has no attribute 'install'
```

Full Traceback: Issue #5 — KeyError: 'Version' (email-validator metadata)

```
File "/opt/nemo_rl_venv/lib/python3.12/site-packages/importlib_metadata/__init__.py", line 514, in version
return self.metadata['Version']
~~~~~~~~~~~~~^^^^^^^^^^^
File "/opt/nemo_rl_venv/lib/python3.12/site-packages/importlib_metadata/_adapters.py", line 101, in __getitem__
raise KeyError(item)
KeyError: 'Version'
```

Full Traceback: Issue #6 — ModuleNotFoundError: No module named 'vllm.logger'

```
File "/opt/nemo-rl/nemo_rl/models/generation/vllm/vllm_worker.py", line 165, in __init__
from vllm.logger import init_logger
ModuleNotFoundError: No module named 'vllm.logger'
```

Full Traceback: Issue #7 — AttributeError: module 'tabulate' has no attribute 'tabulate'

```
File "/opt/nemo_rl_venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 239, in tabulate [repeated 2x across cluster]
(VllmGenerationWorker pid=263199) return tabulate.tabulate(rows, headers=headers) [repeated 2x across cluster]
(VllmGenerationWorker pid=263199) ^^^^^^^^^^^^^^^^^ [repeated 2x across cluster]
(VllmGenerationWorker pid=263199) AttributeError: module 'tabulate' has no attribute 'tabulate' [repeated 2x across cluster]
Error removing placement group PlacementGroupID(b3e63513e3899b493f8bf717b83f01000000): sys.meta_path is None, Python is likely shutting down
Error during policy shutdown: sys.meta_path is None, Python is likely shutting down
```

Full Traceback: Issue #9 — RuntimeError: DeepGEMM backend is not available

```
(VllmGenerationWorker pid=289714) File "/opt/nemo_rl_venv/lib/python3.12/site-packages/vllm/utils/deep_gemm.py", line 85, in _missing [repeated 7x across cluster]
(VllmGenerationWorker pid=289714) raise RuntimeError( [repeated 7x across cluster]
(VllmGenerationWorker pid=289714) RuntimeError: DeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels. [repeated 7x across cluster]
Error removing placement group PlacementGroupID(ebc03bb45a5609a8fc5b6c54b29901000000): sys.meta_path is None, Python is likely shutting down
Error during policy shutdown: sys.meta_path is None, Python is likely shutting down
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.