bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
is fused layernorm really better?
Open
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
https://github.com/pytorch/pytorch/commit/8b87f9a5107e8b3c4f87d5297af698bb55838d81#diff-f12c726e3e8cd2b4768f8984fef27059
I think we don't need to use apex fused layernorm anymore.
torch layernorm is better. what do you think about this?
cc. @stas00 @thomasw21
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.