THUDM / THUDM/slime

ncclUnhandledCudaError: Call to CUDA function failed. Cuda failure 999 'unknown error'

Open
#1,305 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

I tried to run the official example run-qwen3-30B-A3B.sh on 8xH800 with the guidance https://thudm.github.io/slime/examples/qwen3-30B-A3B.html. After several training steps (usually less than 10 steps), the job failed with the following error:

torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.27.3
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 999 'unknown error'

My environment is built with slime 0.2.0.post1, based on the guidance https://github.com/THUDM/slime/blob/763f18d5bd24deb456e863086455983d474ffb75/build_conda.sh (due to some constraints I cannot use Docker).

I also tried to re-run the job for many times, however this problem always exists. I do not observe OOM, and after export NCCL_DEBUG=INFO I cannot find useful information. And this error appears at different functions (such as actor_model.async_train and actor_model.update_weights, see below).

Any suggestions to solve this problem? Thank you in advance.

Location 1: during actor_model.async_train

(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:17] timer.py:24 - Timer wake_up start
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:17] memory_utils.py:41 - [Rank 0] Memory-Usage before wake_up model: {'gpu': '0', 'total_GB': 79.11, 'free_GB': 67.15, 'used_GB': 11.96, 'allocated_GB': 24.83, 'reserved_GB': 25.58}
(MegatronTrainRayActor pid=38724) [2026-01-02 14:56:24] reloadable_process_group.py:152 - Reloading 20 process groups in pid 38724
(MegatronTrainRayActor pid=38724) [2026-01-02 14:56:17] memory_utils.py:41 - [Rank 7] Memory-Usage before wake_up model: {'gpu': '7', 'total_GB': 79.11, 'free_GB': 69.39, 'used_GB': 9.72, 'allocated_GB': 25.36, 'reserved_GB': 25.96} [repeated 7x across cluster]
(MegatronTrainRayActor pid=38724) [2026-01-02 14:56:24] memory_utils.py:41 - [Rank 7] Memory-Usage after wake_up model: {'gpu': '7', 'total_GB': 79.11, 'free_GB': 44.05, 'used_GB': 35.06, 'allocated_GB': 25.36, 'reserved_GB': 25.96}
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:25] timer.py:32 - Timer wake_up end (elapsed: 8.1s)
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:25] timer.py:24 - Timer data_preprocess start
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:25] timer.py:32 - Timer data_preprocess end (elapsed: 0.2s)
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:32] timer.py:32 - Timer train_wait end (elapsed: 163.5s)
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:32] timer.py:24 - Timer train start
(MegatronTrainRayActor pid=38728) [2026-01-02 14:56:28] reloadable_process_group.py:152 - Reloading 20 process groups in pid 38728 [repeated 7x across cluster]
(MegatronTrainRayActor pid=38728) [2026-01-02 14:56:28] memory_utils.py:41 - [Rank 3] Memory-Usage after wake_up model: {'gpu': '3', 'total_GB': 79.11, 'free_GB': 40.44, 'used_GB': 38.67, 'allocated_GB': 25.4, 'reserved_GB': 28.87} [repeated 7x across cluster]
(MegatronTrainRayActor pid=38135) [2026-01-02 14:56:32] timer.py:24 - Timer ref_log_probs start
Traceback (most recent call last):
  File "/root/src/slime/train.py", line 106, in <module>
    train(args)
  File "/root/src/slime/train.py", line 79, in train
    ray.get(actor_model.async_train(rollout_id, rollout_data_ref))
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/worker.py", line 2972, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/worker.py", line 1031, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistBackendError): ray::MegatronTrainRayActor.train() (pid=38729, ip=10.63.230.147, actor_id=8b3190d1c1926ac755dcfaca02000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x7fc281a1b290>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/actor.py", line 311, in train
    return self.train_actor(rollout_id, rollout_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/actor.py", line 354, in train_actor
    self.compute_log_prob(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/actor.py", line 289, in compute_log_prob
    return forward_only(
           ^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/model.py", line 254, in forward_only
    forward_data_store += forward_backward_func(
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 600, in forward_backward_no_pipelining
    output_tensor, num_tokens = forward_step(
                                ^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/pipeline_parallel/schedules.py", line 400, in forward_step
    output_tensor, loss_func = forward_step_func(data_iterator, model)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/model.py", line 220, in forward_step
    output_tensor = model(
                    ^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/distributed/data_parallel_base.py", line 22, in forward
    return self.module(*inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/transformer/module.py", line 237, in forward
    outputs = self.module(*inputs, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/models/gpt/gpt_model.py", line 374, in forward
    self._preprocess(
  File "/root/Megatron-LM/megatron/core/models/gpt/gpt_model.py", line 279, in _preprocess
    decoder_input = self.embedding(input_ids=input_ids, position_ids=position_ids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/models/common/embeddings/language_model_embedding.py", line 111, in forward
    word_embeddings = self.word_embeddings(input_ids)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/tensor_parallel/layers.py", line 289, in forward
    output = reduce_scatter_to_sequence_parallel_region(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/tensor_parallel/mappings.py", line 518, in reduce_scatter_to_sequence_parallel_region
    return _ReduceScatterToSequenceParallelRegion.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/autograd/function.py", line 576, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/tensor_parallel/mappings.py", line 365, in forward
    return _reduce_scatter_along_first_dim(input_, group, input_split_sizes, use_global_buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/Megatron-LM/megatron/core/tensor_parallel/mappings.py", line 182, in _reduce_scatter_along_first_dim
    dist_reduce_scatter_func(output, input_.contiguous(), group=group)
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 4450, in reduce_scatter_tensor
    work = group._reduce_scatter_base(output, input, opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/utils/reloadable_process_group.py", line 221, in _reduce_scatter_base
    return self._fwd("_reduce_scatter_base", *a, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/utils/reloadable_process_group.py", line 182, in _fwd
    return getattr(inner, method)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.27.3
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 999 'unknown error'
mem_info={'gpu': '6', 'total_GB': 79.11, 'free_GB': 42.3, 'used_GB': 36.8, 'allocated_GB': 25.47, 'reserved_GB': 25.91}
mem_info={'gpu': '6', 'total_GB': 79.11, 'free_GB': 42.3, 'used_GB': 36.8, 'allocated_GB': 25.47, 'reserved_GB': 25.91}

Location 2: during actor_model.update_weights

(MegatronTrainRayActor pid=135986) [2026-01-02 15:29:24] timer.py:24 - Timer update_weights start
(MegatronTrainRayActor pid=135986) [2026-01-02 15:29:24] reloadable_process_group.py:152 - Reloading 20 process groups in pid 135986
(MegatronTrainRayActor pid=135986) [2026-01-02 15:29:24] memory_utils.py:41 - [Rank 0] Memory-Usage before update_weights: {'gpu': '0', 'total_GB': 79.11, 'free_GB': 60.0, 'used_GB': 19.11, 'allocated_GB': 23.99, 'reserved_GB': 25.04}
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP6] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP4] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP1] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP5] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP0] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP2] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP7] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24 TP3] Cache flushed successfully!
(SGLangEngine pid=132424) [2026-01-02 15:29:24] INFO:     10.63.230.152:18008 - "GET /flush_cache HTTP/1.1" 200 OK
(MegatronTrainRayActor pid=135986) 
Update weights:   0%|                                                                                    | 0/115 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/root/src/slime/train.py", line 106, in <module>
    train(args)
  File "/root/src/slime/train.py", line 91, in train
    actor_model.update_weights()
  File "/root/src/slime/slime/ray/actor_group.py", line 124, in update_weights
    return ray.get([actor.update_weights.remote() for actor in self._actor_handlers])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/worker.py", line 2972, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/ray/_private/worker.py", line 1031, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(DistBackendError): ray::MegatronTrainRayActor.update_weights() (pid=136503, ip=10.63.230.152, actor_id=906d33aeca7b8a23f030801802000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x7faa391b7a40>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/utils/timer.py", line 78, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/actor.py", line 459, in update_weights
    self.weight_updater.update_weights()
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/update_weight/update_weight_from_tensor.py", line 119, in update_weights
    for hf_named_tensors in self._hf_weight_iterator.get_hf_weight_chunks(megatron_local_weights):
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/update_weight/hf_weight_iterator_direct.py", line 30, in get_hf_weight_chunks
    megatron_full_params = _get_megatron_full_params(megatron_local_param_infos, megatron_local_weights)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/src/slime/slime/backends/megatron_utils/update_weight/hf_weight_iterator_direct.py", line 90, in _get_megatron_full_params
    torch.distributed.broadcast(
  File "/root/src/slime/slime/utils/reloadable_process_group.py", line 55, in new_function
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/conda/envs/my_slime020/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2824, in broadcast
    work = group.broadcast([tensor], opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:77, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.27.3
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 999 'unknown error'
mem_info={'gpu': '7', 'total_GB': 79.11, 'free_GB': 61.93, 'used_GB': 17.18, 'allocated_GB': 24.49, 'reserved_GB': 24.95}
mem_info={'gpu': '7', 'total_GB': 79.11, 'free_GB': 61.93, 'used_GB': 17.18, 'allocated_GB': 24.49, 'reserved_GB': 24.95}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing examples/qwen3-30B-A3B.html with run-qwen3-30B-A3B.sh and NCCL_DEBUG=INFO. Read slime/utils/reloadable_process_group.py, slime/backends/megatron_utils/actor.py, and slime/backends/megatron_utils/model.py around the reported calls, then compare failures in async_train and update_weights. Done means identifying a reproducible cause and documenting or fixing it with validation on the 8xH800 setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.