EleutherAI / EleutherAI/cookbook
Add communication volume calculation script
Open
- Dominant language
- Python
- Stars
- 852
- Forks
- 47
- PR merge metrics
- No merged PRs in 30d
Description
Would be good to model the communication volume in bytes of a given parallelism setup. Situations to model:
- Different parallelism schemes
- ZeRO-1/2/3, ZeRO++
- 3D parallelism
- Activation checkpointing
- Different dtypes
Bonus points:
- % volume breakdown separated by collective
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.