deepspeedai / deepspeedai/DeepSpeed
[BUG] DeepSpeed checkpointing partitioning modifies input tensor shape which cannot be reused
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
Currently, when I enable checkpointing partitioning in stage 3, I find that DeepSpeed's checkpoint() function would modify the input tensor shape when wrapping the function call. Making them unable to be re-used in other forward() calls and throwing tensor shape errors. Stage 3 works fine if I do not enable checkpointing partitioning.
To Reproduce
Run with a conventional network doing for-loop in the forward function like this
from deepspeed.runtime.activation_checkpointing import checkpointing
def forward(self, x, condition):
for layer in self.residual_layers:
if checkpointing.is_configured():
x, skip_connection = checkpointing.checkpoint(layer, x, condition)
else:
x, skip_connection = layer(x, condition)
skip = skip_connection if skip is None else skip_connection + skip
The shape of x and condition would be modified and cannot be reused in the next iteration of the for-loop (while the first will succeed).
Details
I did a bunch of debugging and find that the root cause is that during partitioning, the new input tensors are first flattened here
https://github.com/microsoft/DeepSpeed/blob/25e500e8dd8bfb2fe07fe6e7eb6261bd58e6bd71/deepspeed/runtime/activation_checkpointing/checkpointing.py#L380
And then the original arguments are overridden by their own flattened version here
https://github.com/microsoft/DeepSpeed/blob/25e500e8dd8bfb2fe07fe6e7eb6261bd58e6bd71/deepspeed/runtime/activation_checkpointing/checkpointing.py#L435
I realize a simple fix (although, I have to first apply another fix mentioned in https://github.com/microsoft/DeepSpeed/issues/3779 to get the correct partitioning) may be to replace the above line with arg.data.copy_(inp.view(arg.shape).data). However, it would not work in a multi-GPU context since inp.data only contains one part of the original arg.data.
I think this is a crucial issue if we really want checkpointing partitioning to work in DeepSpeed. Without it, I don't see any benefit in replacing the native Pytorch checkpointing with this non-partitioned one.
### Tasks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read deepspeed/runtime/activation_checkpointing/checkpointing.py around lines 380 and 435, then reproduce the issue with the provided looping forward method, stage 3, and checkpointing partitioning enabled. Trace how flattened inputs replace the original tensors and verify a fix preserves reusable tensor shapes in both single- and multi-GPU runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100