deepspeedai / deepspeedai/DeepSpeed

[BUG]deepspeed>=v0.17.3 caused an error in megatron's `initialize_model_parallel`

Open
#7,470 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

/core/parallel_state.py", line 714, in initialize_model_parallel
[rank0]:     group = torch.distributed.new_group(
[rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
[rank0]:     func_return = func(*args, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 5065, in new_group
[rank0]:     return _new_group_with_tag(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 5155, in _new_group_with_tag
[rank0]:     pg, pg_store = _new_process_group_helper(
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2074, in _new_process_group_helper
[rank0]:     eager_backend.eager_connect_single_device(device_id)
[rank0]: torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:210, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.2
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 2 'out of memory'
[rank0]:[W807 20:14:01.469010810 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at /core/parallel_state.py around line 714, where initialize_model_parallel calls torch.distributed.new_group, and compare behavior with DeepSpeed v0.17.3 and later. Reproduce the traceback under the reported Python 3.11, CUDA 12.4, PyTorch 2.7, and NCCL 2.26.2 environment with NCCL_DEBUG=INFO; done means identifying the regression and confirming a fix without the CUDA out-of-memory error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.