muon optimizer support
Open
algorithm
community-request
r0.6.0
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
[muon optimizer](https://github.com/KellerJordan/Muon?tab=readme-ov-file) has attracted lots of interests in the community and is currently WIP in mcore. Also, it has been reported the model performance is even better if the same optimizer is used both in pretraining and post-training stages.
existing effort in distributed mcore which will be merged as part of megatron-core
Contributor guide
Assessment
This issue has not been assessed yet.