bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
Implement Gradient Noise Scale monitoring
Open
arch&scale
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
Follow appendix A.1 https://arxiv.org/pdf/1812.06162.pdf to implement monitoring of gradient noise scale and add it to the tensorboard log.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.