[feat] GTP+moe-single-weight
Open
nemotron
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
Add `moe-single-weight` support for GTP.
Precondition of `moe-single-weight` feature:
- TE: https://github.com/NVIDIA/TransformerEngine/pull/3224
- Mcore: https://github.com/NVIDIA/Megatron-LM/pull/6000
Contributor guide
Assessment
This issue has not been assessed yet.