NVIDIA-NeMo / NVIDIA-NeMo/RL

NCCL Failure on Out-of-Box Colocated GRPO Examples

Open
#1,240 2 comments 0 reactions 1 assignee Claimed by @joyang-nv View on GitHub
bug community-request external t-dependency t-pytdensor waiting-on-maintainers x-ucb
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

Running `run_grpo_math.py` on any of the single-node example configs, with an override to 4 `gpus_per_node` leads to an NCCL error during setup, but runs without failure if `gpus_per_node` is 2. For example, here is end of the error trace from running `grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml` with `gpus_per_node` set to 4.

```
Traceback (most recent call last):
File "/home/neil/nemo/nemo-rl/examples/run_grpo_math.py", line 235, in
main()
File "/home/neil/nemo/nemo-rl/examples/run_grpo_math.py", line 189, in main
) = setup(config, tokenizer, dataset, val_dataset)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/nemo_rl/algorithms/grpo.py", line 440, in setup
state_dict_info = policy.prepare_refit_info()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/nemo_rl/models/policy/lm_policy.py", line 554, in prepare_refit_info
results = ray.get(futures)
^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/worker.py", line 2822, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/neil/nemo/nemo-rl/.venv/lib/python3.12/site-packages/ray/_private/worker.py", line 932, in get_objects
raise value
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: DTensorPolicyWorker
actor_id: 6565bac7760fcfb848d7e84101000000
pid: 678173
name: lm_policy-0-0
namespace: b1901ee7-9701-4525-89d9-22b923524153
ip: 169.229.192.168
The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
The actor never ran - it was cancelled before it started running.
```

**Steps/Code to reproduce bug**

Clone NemoRL, install libibverbs-dev.

Run `run_grpo_math.py` on `grpo-qwen2.5-math-1.5b-instruct-1n8g-fsdp2tp1.v3.yaml` with `gpus_per_node` set to 4.

**Expected behavior**

GRPO loop should begin successfully, instead fails during setup.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.