modelscope / modelscope/ms-swift

CUDA out of memory in multi-node (4× nodes × 4× A100) training Qwen3-Embedding-4B with DeepSpeed ZeRO-3

Open
#6,301 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Training an embedding model (Qwen3-Embedding-4B) with ms-swift / transformers / deepspeed across 4 nodes × 4 A100 80GB GPUs (16 GPUs total) intermittently hits:

  • CUDA out of memory during backward (inside SDPA attention) on one rank, followed by
  • global NCCL watchdog timeouts / SIGABRT on other ranks.


Train:   8%|▊         | 40/500 [15:18<2:32:54, 19.94s/it]
Val: 100%|██████████| 20/20 [00:22<00:00,  1.12s/it]
[rank11]: Traceback (most recent call last):
[rank11]:   File "./ms-swift/train_embedding_ms_swift.py", line 46, in <module>
[rank11]:     result = sft_main(args)
[rank11]:   File ".ms-swift/swift/llm/train/sft.py", line 341, in sft_main
[rank11]:     return SwiftSft(args).main()
[rank11]:   File ./ms-swift/swift/llm/base.py", line 49, in main
[rank11]:     result = self.run()
[rank11]:   File "./ms-swift/swift/llm/train/sft.py", line 196, in run
[rank11]:     return self.train(trainer)
[rank11]:   File "./ms-swift/swift/llm/train/sft.py", line 244, in train
[rank11]:     trainer.train(trainer.args.resume_from_checkpoint)
[rank11]:   File "./ms-swift/swift/trainers/trainers.py", line 57, in train
[rank11]:     return super().train(*args, **kwargs)
[rank11]:   File "./ms-swift/swift/trainers/mixin.py", line 801, in train
[rank11]:     res = super().train(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/trainer.py", line 2325, in train
[rank11]:     return inner_training_loop(
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank11]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/trainer.py", line 4071, in training_step
[rank11]:     self.accelerator.backward(loss, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/accelerate/accelerator.py", line 2732, in backward
[rank11]:     self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 270, in backward
[rank11]:     self.engine.backward(loss, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank11]:     ret_val = func(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2356, in backward
[rank11]:     self._do_optimizer_backward(loss, retain_graph)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2297, in _do_optimizer_backward
[rank11]:     self.optimizer.backward(loss, retain_graph=retain_graph)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank11]:     ret_val = func(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 2335, in backward
[rank11]:     self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 65, in backward
[rank11]:     scaled_loss.backward(retain_graph=retain_graph)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/_tensor.py", line 625, in backward
[rank11]:     torch.autograd.backward(
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/autograd/__init__.py", line 354, in backward
[rank11]:     _engine_run_backward(
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/autograd/graph.py", line 841, in _engine_run_backward
[rank11]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/autograd/function.py", line 315, in apply
[rank11]:     return user_fn(self, *args)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 302, in backward
[rank11]:     outputs = ctx.run_function(*detached_inputs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank11]:     return self._call_impl(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1881, in _call_impl
[rank11]:     return inner()
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1829, in inner
[rank11]:     result = forward_call(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
[rank11]:     return func(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 260, in forward
[rank11]:     hidden_states, _ = self.self_attn(
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
[rank11]:     return self._call_impl(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1881, in _call_impl
[rank11]:     return inner()
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1829, in inner
[rank11]:     result = forward_call(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
[rank11]:     return func(*args, **kwargs)
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 216, in forward
[rank11]:     attn_output, attn_weights = attention_interface(
[rank11]:   File "./.conda/envs/swift/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward
[rank11]:     attn_output = torch.nn.functional.scaled_dot_product_attention(
[rank11]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.35 GiB. GPU 3 has a total capacity of 63.42 GiB of which 1.70 GiB is free. Process 2521735 has 474.00 MiB memory in use. Including non-PyTorch memory, this process has 60.30 GiB memory in use. Process 2521734 has 474.00 MiB memory in use. Process 2521733 has 474.00 MiB memory in use. Of the allocated memory 50.94 GiB is allocated by PyTorch, and 8.56 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
W1026 03:51:31.222000 2521679 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2521733 closing signal SIGTERM
W1026 03:51:31.222000 2521679 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2521734 closing signal SIGTERM
W1026 03:51:31.222000 2521679 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 2521735 closing signal SIGTERM
E1026 03:51:31.737000 2521679 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 3 (pid: 2521736) of binary: ./.conda/envs/swift/bin/python
Traceback (most recent call last):
  File "./.conda/envs/swift/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "./.conda/envs/swift/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 940, in <module>
    main()
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
train_embedding_ms_swift.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-10-26_03:51:31
  host      : lrdn0957.leonardo.local
  rank      : 11 (local_rank: 3)
  exitcode  : 1 (pid: 2521736)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
srun: error: lrdn0957: task 2: Exited with exit code 1
[rank12]:[E1026 04:01:29.391852812 ProcessGroupNCCL.cpp:683] [Rank 12] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600003 milliseconds before timing out.
[rank12]:[E1026 04:01:29.394972783 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 12]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank12]:[E1026 04:01:29.394982929 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank12]:[E1026 04:01:29.395011304 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 12] First PG on this rank to signal dumping.

Train:   8%|▊         | 41/500 [15:39<3:29:41, 27.41s/it]
Train:   8%|▊         | 42/500 [15:53<2:59:03, 23.46s/it]
Train:   9%|▊         | 43/500 [16:18<3:02:16, 23.93s/it]
Train:   9%|▉         | 44/500 [16:33<2:40:25, 21.11s/it]
Train:   9%|▉         | 45/500 [16:49<2:30:07, 19.80s/it]
Train:   9%|▉         | 46/500 [17:26<3:08:24, 24.90s/it]
Train:   9%|▉         | 47/500 [17:41<2:45:40, 21.94s/it][rank2]:[E1026 04:01:29.440969438 ProcessGroupNCCL.cpp:683] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600019 milliseconds before timing out.
[rank5]:[E1026 04:01:29.462422460 ProcessGroupNCCL.cpp:683] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600019 milliseconds before timing out.
[rank2]:[E1026 04:01:29.444112953 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 2]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank2]:[E1026 04:01:29.444124613 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E1026 04:01:29.444150962 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 2] First PG on this rank to signal dumping.
[rank5]:[E1026 04:01:29.465823296 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 5]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank5]:[E1026 04:01:29.465838221 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank5]:[E1026 04:01:29.465863880 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 5] First PG on this rank to signal dumping.
[rank4]:[E1026 04:01:29.476086734 ProcessGroupNCCL.cpp:683] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600033 milliseconds before timing out.
[rank4]:[E1026 04:01:29.476192382 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 4]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank4]:[E1026 04:01:29.476199047 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank4]:[E1026 04:01:29.476223774 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 4] First PG on this rank to signal dumping.
[rank14]:[E1026 04:01:29.433054081 ProcessGroupNCCL.cpp:683] [Rank 14] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600046 milliseconds before timing out.
[rank14]:[E1026 04:01:29.433156908 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 14]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank14]:[E1026 04:01:29.433162790 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank14]:[E1026 04:01:29.433189268 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 14] First PG on this rank to signal dumping.
[rank15]:[E1026 04:01:29.444400831 ProcessGroupNCCL.cpp:683] [Rank 15] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600056 milliseconds before timing out.
[rank15]:[E1026 04:01:29.444503938 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 15]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank15]:[E1026 04:01:29.444510727 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank15]:[E1026 04:01:29.444534690 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 15] First PG on this rank to signal dumping.
[rank13]:[E1026 04:01:29.446171636 ProcessGroupNCCL.cpp:683] [Rank 13] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600059 milliseconds before timing out.
[rank13]:[E1026 04:01:29.446279064 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 13]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank13]:[E1026 04:01:29.446285324 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank13]:[E1026 04:01:29.446309113 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 13] First PG on this rank to signal dumping.
[rank1]:[E1026 04:01:29.489389048 ProcessGroupNCCL.cpp:683] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600066 milliseconds before timing out.
[rank1]:[E1026 04:01:29.489489018 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 1]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank1]:[E1026 04:01:29.489494960 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank1]:[E1026 04:01:29.489521185 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 1] First PG on this rank to signal dumping.
[rank0]:[E1026 04:01:29.500842480 ProcessGroupNCCL.cpp:683] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600079 milliseconds before timing out.
[rank0]:[E1026 04:01:29.500941629 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 0]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank0]:[E1026 04:01:29.500953846 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank0]:[E1026 04:01:29.500973224 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 0] First PG on this rank to signal dumping.
[rank3]:[E1026 04:01:29.510593654 ProcessGroupNCCL.cpp:683] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600087 milliseconds before timing out.
[rank3]:[E1026 04:01:29.510702382 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 3]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank3]:[E1026 04:01:29.510709124 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank3]:[E1026 04:01:29.510734532 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 3] First PG on this rank to signal dumping.
[rank7]:[E1026 04:01:29.531607250 ProcessGroupNCCL.cpp:683] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600088 milliseconds before timing out.
[rank7]:[E1026 04:01:29.531707064 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 7]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank7]:[E1026 04:01:29.531712538 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank7]:[E1026 04:01:29.531735035 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 7] First PG on this rank to signal dumping.
[rank6]:[E1026 04:01:29.536458714 ProcessGroupNCCL.cpp:683] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=55485, OpType=_REDUCE_SCATTER_BASE, NumelIn=2560, NumelOut=160, Timeout(ms)=600000) ran for 600094 milliseconds before timing out.
[rank6]:[E1026 04:01:29.536559407 ProcessGroupNCCL.cpp:2241] [PG ID 1 PG GUID 1 Rank 6]  failure detected by watchdog at work sequence id: 55485 PG status: last enqueued work: 55487, last completed work: 55484
[rank6]:[E1026 04:01:29.536564960 ProcessGroupNCCL.cpp:730] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank6]:[E1026 04:01:29.536585733 ProcessGroupNCCL.cpp:2573] [PG ID 1 PG GUID 1 Rank 6] First PG on this rank to signal dumping.
[rank13]:[E1026 04:01:29.921309618 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 13] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank14]:[E1026 04:01:29.921321373 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 14] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank15]:[E1026 04:01:29.921357545 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 15] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank12]:[E1026 04:01:29.921412997 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 12] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank15]:[E1026 04:01:29.922296922 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 15] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank14]:[E1026 04:01:29.922296938 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 14] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank13]:[E1026 04:01:29.922298054 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 13] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank12]:[E1026 04:01:29.922306748 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 12] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank5]:[E1026 04:01:29.061648531 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 5] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank6]:[E1026 04:01:29.061658494 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 6] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank7]:[E1026 04:01:29.061701855 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 7] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank4]:[E1026 04:01:29.061704611 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 4] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank5]:[E1026 04:01:29.062683857 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 5] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank6]:[E1026 04:01:29.062683328 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 6] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank7]:[E1026 04:01:29.062690119 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 7] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank4]:[E1026 04:01:29.062699892 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 4] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank0]:[E1026 04:01:30.104968587 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 0] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank3]:[E1026 04:01:30.104985335 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 3] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank1]:[E1026 04:01:30.105005208 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 1] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank2]:[E1026 04:01:30.105007493 ProcessGroupNCCL.cpp:1858] [PG ID 0 PG GUID 0(default_pg) Rank 2] Received a dump signal due to a collective timeout from this local rank and we will try our best to dump the debug info. Last enqueued NCCL work: 79209, last completed NCCL work: 79196.This is most likely caused by incorrect usages of collectives, e.g., wrong sizes used across ranks, the order of collectives is not same for all ranks or the scheduled collective, for some reason, didn't run. Additionally, this can be caused by GIL deadlock or other reasons such as network errors or bugs in the communications library (e.g. NCCL), etc. 
[rank1]:[E1026 04:01:30.105691758 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank0]:[E1026 04:01:30.105691566 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank2]:[E1026 04:01:30.105693140 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 2] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank3]:[E1026 04:01:30.105696218 ProcessGroupNCCL.cpp:1575] [PG ID 0 PG GUID 0(default_pg) Rank 3] ProcessGroupNCCL preparing to dump debug info. Include stack trace: 1
[rank14]:[F1026 04:09:29.931090093 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 14] [PG ID 0 PG GUID 0(default_pg) Rank 14] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank12]:[F1026 04:09:29.931114609 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 12] [PG ID 0 PG GUID 0(default_pg) Rank 12] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank15]:[F1026 04:09:29.931165254 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 15] [PG ID 0 PG GUID 0(default_pg) Rank 15] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank13]:[F1026 04:09:29.931221912 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 13] [PG ID 0 PG GUID 0(default_pg) Rank 13] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank5]:[F1026 04:09:29.067921917 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 5] [PG ID 0 PG GUID 0(default_pg) Rank 5] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank7]:[F1026 04:09:29.067927312 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 7] [PG ID 0 PG GUID 0(default_pg) Rank 7] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank6]:[F1026 04:09:29.068044893 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 6] [PG ID 0 PG GUID 0(default_pg) Rank 6] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank4]:[F1026 04:09:29.068408728 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 4] [PG ID 0 PG GUID 0(default_pg) Rank 4] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
E1026 04:09:29.991000 3075410 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: -6) local_rank: 0 (pid: 3075464) of binary: ./.conda/envs/swift/bin/python
Traceback (most recent call last):
  File "./.conda/envs/swift/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "./.conda/envs/swift/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 940, in <module>
    main()
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "./.conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

========================================================
[rank0]:[F1026 04:09:30.110319317 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 0] [PG ID 0 PG GUID 0(default_pg) Rank 0] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank2]:[F1026 04:09:30.110319907 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 2] [PG ID 0 PG GUID 0(default_pg) Rank 2] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank3]:[F1026 04:09:30.110319754 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 3] [PG ID 0 PG GUID 0(default_pg) Rank 3] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
[rank1]:[F1026 04:09:30.110344019 ProcessGroupNCCL.cpp:1600] [PG ID 0 PG GUID 0(default_pg) Rank 1] [PG ID 0 PG GUID 0(default_pg) Rank 1] Terminating the process after attempting to dump debug info, due to collective timeout or exception.
E1026 04:09:30.101000 2904014 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: -6) local_rank: 0 (pid: 2904068) of binary:  /.conda/envs/swift/bin/python
Traceback (most recent call last):
  File " /.conda/envs/swift/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/.conda/envs/swift/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 940, in <module>
    main()
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File ".conda/envs/swift/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
Training script (minimal)
# train_embedding_ms_swift.py
import os
from swift.llm import sft_main, TrainArguments

BASE_MODEL = "./Qwen3-Embedding-4B"
LOCAL_REPO = None
TRAIN_TYPE = "full"

args = TrainArguments(
    model=BASE_MODEL,
    model_type="qwen3_emb",
    task_type="embedding",
    train_type=TRAIN_TYPE,
    dataset=["./reformated_ms_swift_embedding_data_without_think.jsonl"],
    split_dataset_ratio=0.05,
    num_train_epochs=5,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=6e-6,
    loss_type="infonce",
    do_train=True,
    dataloader_drop_last=True,
    eval_strategy="steps",
    eval_steps=20,
    save_steps=100,
    save_total_limit=5,
    logging_steps=10,

    # crucial bits:
    use_hf=True,                      
    local_repo_path=LOCAL_REPO,      
    check_model=False,
    deepspeed="zero3",               
    output_dir="output/embedding_model_full",
)

if __name__ == "__main__":
    result = sft_main(args)
    print("Done. Saved to:", args.output_dir)

srun --jobid $SLURM_JOBID bash -c '
    python -m torch.distributed.run \
        --nproc_per_node=\$GPUS_PER_NODE \
        --nnodes=\$SLURM_NNODES \
        --node_rank=\$SLURM_PROCID \
        --master_addr=\$MASTER_ADDR \
        --master_port=\$MASTER_PORT \
        train_embedding_ms_swift.py
'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with train_embedding_ms_swift.py and the traceback around the backward pass in Qwen3 SDPA attention under DeepSpeed ZeRO-3. Reproduce the 4-node, 16-GPU training run and compare the CUDA out-of-memory failure with the subsequent NCCL watchdog timeout. Done means identifying a reproducible cause and confirming training completes without either failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.