deepspeedai / deepspeedai/DeepSpeed
[BUG]DeepSpeed zero3 use more gpu memory than zero2
Open
@umchand is already working on this.
Since Apr 1, 2024.
bug
training
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
- DeepSpeed zero3 use more gpu memory than zero2.
- Ulysess performance problem.
To Reproduce
Steps to reproduce the behavior:
just use ulysess sample code: https://github.com/microsoft/Megatron-DeepSpeed/blob/main/examples_deepspeed/sequence_parallel/ds_pretrain_gpt_1.3B_seq_parallel_32k.sh
change the model size into 7b, use 8xA100-80g, gbs=16, mbs=1, dp=2, sp=4
for deepspeed: dp2+sp4:
- zero3, recompute: 52.8G, 13000 ms
- zero2, recompute: 43.8G, 13194 ms
- zero3, no-recompute: 75.2G, 10329 ms
- zero2, no-recompute: 62.3G, 10050 ms
for tensor parallel: dp2+tp4:
- recompute: 32.5G, 16128 ms
- no-recompute: 45G, 12109 ms
I'm so confused that:
- why zero3 use more gpu memory than zero2?
- it seems if do not use checkpoint-activations, use the same dp, ulysess will oom earlier than megatron tp?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.