deepspeedai / deepspeedai/DeepSpeed
[BUG]Deepspeed (v0.15.4 ～v0.16.9) Zero3 training performance is slow，compare than v0.13.1
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
When I used a single machine with eight A100 cards and Ascend NPU for model training. Due to the version upgrade of the supporting software, I needed to upgrade the DeepSpeed version from 0.13.1 to 0.15.4 or above. However, after upgrading DeepSpeed, when using the DeepSpeed Zero3 configuration, the training performance deteriorated by about 10% compared to version 0.13.1. This issue persisted even in version 0.16.9. But when using Zero2 or Zero1, there was no such problem.
May I ask if there is any way to avoid or solve the performance degradation of Zero3 during the upgrade process of DeepSpeed?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Start by reproducing Zero3 training on one machine with eight A100 cards and Ascend NPU, comparing versions 0.13.1 and 0.15.4–0.16.9 alongside Zero1 and Zero2; done means explaining and resolving the reported roughly 10% regression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100