docs or demos to illustrate the key features
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
Congrats on the release! Some of the features are so cool they feel like black magic.
Would it be possible to explain the key techniques behind those features, or provide a tutorial/demo so users can reproduce the claimed results?
In particularly, I am interested in the claim
> Memory-efficient design: Train 200B MoE models on 64k sequence lengths without sequence parallelism through advanced memory optimization techniques
It sounds very challenging, unless there’s aggressive offloading and recomputation and may suffer from slow iteration speed.
Contributor guide
Assessment
This issue has not been assessed yet.