deepspeedai / deepspeedai/DeepSpeed

[BUG] `reduce_bucket_size` isn't validated against the model size

Open
#4,875 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug

When a model is small and the reduce_bucket_size is larger this happens:

  File "/mnt/nvme0/code/huggingface/accelerate-master/src/accelerate/utils/deepspeed.py", line 167, in backward
    self.engine.backward(loss, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1981, in backward
    self.allreduce_gradients()
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1902, in allreduce_gradients
    self.optimizer.overlapping_partition_gradients_reduce_epilogue()
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1098, in overlapping_partition_gradients_reduce_epilogue
    self.independent_gradient_partition_epilogue()
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1075, in independent_gradient_partition_epilogue
    self.__reduce_and_partition_ipg_grads()
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1188, in __reduce_and_partition_ipg_grads
    assert len(set(p.ds_id for p in self.params_in_ipg_bucket)) == len(self.params_in_ipg_bucket)
AssertionError
    self.__reduce_and_partition_ipg_grads()
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/stas/anaconda3/envs/py39-pt21/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 1188, in __reduce_and_partition_ipg_grads
    assert len(set(p.ds_id for p in self.params_in_ipg_bucket)) == len(self.params_in_ipg_bucket)
AssertionError
  1. it probably could check at launch time that the buffer isn't larger than the model size
  2. in any case the assert needs to tell the user what's wrong and how they can fix the problem - the current message-less assert isn't doing that.

Thanks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in deepspeed/runtime/zero/stage3.py at __reduce_and_partition_ipg_grads, then trace how reduce_bucket_size is handled before training starts. Done means oversized values are detected early and the existing assertion reports the cause and a way to fix it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.