deepspeedai / deepspeedai/DeepSpeed

[BUG] The same program runs fine with v0.17.5, but fails with v0.17.6. Under the zero2 configuration

Open
#7,629 1 comment 2 reactions 1 assignee View on GitHub

@loadams is already working on this.

Since Oct 16, 2025.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
rank2: ret_val = func(*args, **kwargs)
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2324, in backward
rank2: self._backward_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2260, in _backward_epilogue
rank2: self.allreduce_gradients()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
rank2: ret_val = func(*args, **kwargs)
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2211, in allreduce_gradients
rank2: self.optimizer.overlapping_partition_gradients_reduce_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 946, in overlapping_partition_gradients_reduce_epilogue
rank2: self.independent_gradient_partition_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 860, in independent_gradient_partition_epilogue
rank2: for accumulated_grad, new_avg_grad in zip(self.all_grad_tensors[i], avg_new):
rank2: ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rank2: TypeError: 'NoneType' object is not iterable

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.