NVIDIA / NVIDIA/Megatron-LM

[QUESTION] plan to implement zero bubble pipeline or dual pipeline and MoE comm-comp overlapping

Open
#1,399 3 comments 10 reactions 0 assignees View on GitHub
community-request module: moe question
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

Hi team,
It looks like you're busy reproducing DeepSeek V3 features such as device-routed load balancing, aux-free load balancing, and etc.
I'm just wondering if you have any plans to implement zero-bubble pipelining or dual pipelining and MoE communication-computation overlapping, as these are crucial for optimizing MoE training.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.