deepspeedai / deepspeedai/DeepSpeed
[BUG]deepspeed>=v0.17.3 caused an error in megatron's `initialize_model_parallel`
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
/core/parallel_state.py", line 714, in initialize_model_parallel
[rank0]: group = torch.distributed.new_group(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 95, in wrapper
[rank0]: func_return = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 5065, in new_group
[rank0]: return _new_group_with_tag(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 5155, in _new_group_with_tag
[rank0]: pg, pg_store = _new_process_group_helper(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/e/conda-py311-cu124-torch27/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 2074, in _new_process_group_helper
[rank0]: eager_backend.eager_connect_single_device(device_id)
[rank0]: torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.cpp:210, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.2
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 2 'out of memory'
[rank0]:[W807 20:14:01.469010810 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at /core/parallel_state.py around line 714, where initialize_model_parallel calls torch.distributed.new_group, and compare behavior with DeepSpeed v0.17.3 and later. Reproduce the traceback under the reported Python 3.11, CUDA 12.4, PyTorch 2.7, and NCCL 2.26.2 environment with NCCL_DEBUG=INFO; done means identifying the regression and confirming a fix without the CUDA out-of-memory error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100