[QUESTION] plan to implement zero bubble pipeline or dual pipeline and MoE comm-comp overlapping
Open
community-request
module: moe
question
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Hi team,
It looks like you're busy reproducing DeepSeek V3 features such as device-routed load balancing, aux-free load balancing, and etc.
I'm just wondering if you have any plans to implement zero-bubble pipelining or dual pipelining and MoE communication-computation overlapping, as these are crucial for optimizing MoE training.
Contributor guide
Assessment
This issue has not been assessed yet.