intelligent-machine-learning / intelligent-machine-learning/atorch
Implement DualPipe training for moe model
- Dominant language
- Python
- Stars
- 62
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
[DualPipe](https://github.com/deepseek-ai/DualPipe) is an efficient pipeline parallel implementation for moe models. This feature request is to implement moe model training using DualPipe .
**Describe the solution you'd like**
Given an example of moe dualpipe training and put it under [examples/moe_dualpipe/](https://github.com/intelligent-machine-learning/atorch/tree/main/examples)
Useful modules or APIs can put in suitable atorch path, such as [atorch/modules/moe/](https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/modules/moe)
Use All2All for moe token communication first, then consider using deep_ep dispatch/combine for optimization.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.