deepspeedai / deepspeedai/DeepSpeed
zero-3 cpuadam is so slow
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
training setup is like
- cuda 12.1 / torch 2.5 / deepspeed 0.14.5
- 4 node (a100-80gb)
- qwen72b
- zero-3
- offload optims
- offload params
- approximately 8192 batch tokens as inputs
you can see cpuadam.step() is very slow and it causes very low MFU.
whereas, when i test the model with zero-3 without offload, it took only 90ms (i know for sure GPU implementation is way faster but...).
i wonder if it's acceptable values for current deepspeed or my skill issue first,
and current cpuadam is fused or not.
and if it's not fused with backward, i want to ask is there any plan for fused cpuadam.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the reported CPUAdam step time in the issue’s screenshots and the listed CUDA, PyTorch, DeepSpeed, ZeRO-3, and offload setup. Compare the approximately 8192-token, four-node A100 configuration with the reported 90 ms no-offload run. Done means determining whether the slowdown is expected and whether CPUAdam is fused with backward or has a planned fused implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100