deepspeedai / deepspeedai/DeepSpeed
[BUG] stage3 cost more gpu vram than stage 2,why?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
i'm using the deepspeed to tran the animat anyone.i want to use stage3 to reduce the cost of gpu vram. theoretically，stage3 will reduce more memory than stage2. but in fact, it didn't
To Reproduce
use this repo
use deepspeed with accelerate to train the stage 2（the 2nd stage of this model not the stage 2 of deepspeed). and you will find the deepspeed stage3 will cost more memory than deepspeed stage2. besides, the cpu offload has no help to reduce memory,i was confused
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Moore-AnimateAnyone repository and reproduce training with DeepSpeed stage 2, stage 3, and CPU offload through Accelerate. Compare GPU memory usage and record the DeepSpeed, Accelerate, model, and hardware configuration. Done means the difference is explained with a reproducible diagnosis or a confirmed fix path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100