deepspeedai / deepspeedai/DeepSpeed
[BUG] `reduce_bucket_size` isn't validated against the model size
Open
Nobody has claimed this yet.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
When a model is small and the reduce_bucket_size is larger this happens:
File "/mnt/nvme0/code/huggingface/accelerate-master/src/accelerate/utils/deepspeed.py", line 167, in backward
self.engine.backward(loss, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1981, in backward
self.allreduce_gradients()
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1902, in allreduce_gradients
self.optimizer.overlapping_partition_gradients_reduce_epilogue()
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1098, in overlapping_partition_gradients_reduce_epilogue
self.independent_gradient_partition_epilogue()
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1075, in independent_gradient_partition_epilogue
self.__reduce_and_partition_ipg_grads()
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1188, in __reduce_and_partition_ipg_grads
assert len(set(p.ds_id for p in self.params_in_ipg_bucket)) == len(self.params_in_ipg_bucket)
AssertionError
self.__reduce_and_partition_ipg_grads()
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1188, in __reduce_and_partition_ipg_grads
assert len(set(p.ds_id for p in self.params_in_ipg_bucket)) == len(self.params_in_ipg_bucket)
AssertionError
- it probably could check at launch time that the buffer isn't larger than the model size
- in any case the assert needs to tell the user what's wrong and how they can fix the problem - the current message-less assert isn't doing that.
Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in deepspeed/runtime/zero/stage3.py at __reduce_and_partition_ipg_grads, then trace how reduce_bucket_size is handled before training starts. Done means oversized values are detected early and the existing assertion reports the cause and a way to fix it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100