deepspeedai / deepspeedai/DeepSpeed
[BUG] The same program runs fine with v0.17.5, but fails with v0.17.6. Under the zero2 configuration
@loadams is already working on this.
Since Oct 16, 2025.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
rank2: ret_val = func(*args, **kwargs)
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2324, in backward
rank2: self._backward_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2260, in _backward_epilogue
rank2: self.allreduce_gradients()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
rank2: ret_val = func(*args, **kwargs)
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/engine.py", line 2211, in allreduce_gradients
rank2: self.optimizer.overlapping_partition_gradients_reduce_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 946, in overlapping_partition_gradients_reduce_epilogue
rank2: self.independent_gradient_partition_epilogue()
rank2: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
rank2: File "/mnt/data/anaconda3/envs/optimizer/lib/python3.13/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 860, in independent_gradient_partition_epilogue
rank2: for accumulated_grad, new_avg_grad in zip(self.all_grad_tensors[i], avg_new):
rank2: ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rank2: TypeError: 'NoneType' object is not iterable
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.