intelligent-machine-learning / intelligent-machine-learning/atorch

Implement DualPipe training for moe model

Open
#9 2 comments 0 reactions 1 assignee Claimed by @xhx1022 View on GitHub
Dominant language
Python
Stars
62
Forks
6
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
[DualPipe](https://github.com/deepseek-ai/DualPipe) is an efficient pipeline parallel implementation for moe models. This feature request is to implement moe model training using DualPipe .

**Describe the solution you'd like**
Given an example of moe dualpipe training and put it under [examples/moe_dualpipe/](https://github.com/intelligent-machine-learning/atorch/tree/main/examples)
Useful modules or APIs can put in suitable atorch path, such as [atorch/modules/moe/](https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/modules/moe)

Use All2All for moe token communication first, then consider using deep_ep dispatch/combine for optimization.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.