[Feature Request] Need ZeRo-1/2 to cooperate with PP+TP+DP. Which may more faster than FSDP sometimes.
Open
- Dominant language
- Python
- Stars
- 560
- Forks
- 72
- PR merge metrics
- No merged PRs in 30d
Description
For example in single A100 machine. Llama2 13B training speed with TP2 DP 4 + Zero1 is more faster than FSDP.
Contributor guide
Assessment
This issue has not been assessed yet.