deepspeedai / deepspeedai/DeepSpeed
[BUG] gradient_accumulation_steps = 2 uses significantly more memory compared to gradient_accumulation_steps = 1 when using pipeline parallelism
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
When using pipeline parallelism + stage 1, we ecounter CUDA OOM errors if gradient_accumulation_steps=2 but everything runs fine with gradient_accumulation_steps=1. My understanding is that gradient_accumulation_steps should not impact memory comsumption.
Expected behavior
gradient_accumulation_steps should not impact memory comsumption.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported pipeline-parallel training case with stage 1 at gradient_accumulation_steps=1 and 2, and compare CUDA memory use and OOM behavior. Done means identifying and correcting the cause so changing the accumulation steps does not significantly increase memory consumption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100