Lightning-AI / Lightning-AI/pytorch-lightning

Deepspeed Startegy doesn't set num_checkpoints while using activation partitions

Open
#20,329 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug strategy: deepspeed ver: 2.4.x
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

### Bug description

When training with DeepSpeed and configuring the ZeRO Stage 3 strategy, if activation partitioning is enabled along with contiguous_checkpointing, you may encounter an "index out of range" error related to contiguous_data_buffers. This issue arises because, during the creation of the activation partition configuration, the num_checkpoints parameter is not passed. As a result, DeepSpeed uses the global variable num_layers with its default value of False, which leads to the incorrect creation of an empty contiguous_data_buffers.

### What version are you seeing the problem on?

v2.4

### How to reproduce the bug

_No response_

### Error messages and logs

```
[rank0]: File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 574, in apply
[rank0]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank0]: File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 557, in forward
[rank0]: inputs = partition_activations(args, CPU_CHECKPOINT, CONTIGUOUS_CHECKPOINTING)
[rank0]: File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 421, in partition_activations
[rank0]: contiguous_data_buffers[i][data_offsets[i]].data[range(
[rank0]: IndexError: list index out of range
```

### Environment

Current environment

```
#- PyTorch Lightning Version: 2.4.0
#- PyTorch Version: 2.4.1
#- Python version: 3.10.6
#- OS: Ubuntu-22.04
#- CUDA version: 12.1
#- GPU models and configuration: A100
#- How you installed Lightning(`conda`, `pip`, source): pip insatll
```

### More info

_No response_

cc @lantiga

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the ZeRO Stage 3 strategy's activation partition configuration into DeepSpeed's activation_checkpointing/checkpointing.py, especially forward and partition_activations from the reported stack trace. Reproduce with activation partitioning and contiguous_checkpointing enabled, then verify that the configuration receives num_checkpoints and no longer raises IndexError for contiguous_data_buffers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.