deepseek-ai / deepseek-ai/DeepEP

AllToAll performance not improving

Open
#59 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hello DeepEP developers, appreciate the solid works!
I conducted an all-to-all communication performance comparison experiment on 8 H100 nodes, using [Megatron-LM's DeepEP Integration](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core/transformer/moe#performance-optimizations).
To eliminate the impact of pipeline parallelism bubbles and focus on comm performances, I use the first pipeline stage (8 layers with first-3-dense) of the DeepSeek V3 model only, instead of the full model.
I ran pretraining experiments with DeepEP enabled/disabled(other configurations are identical), and the throughputs were roughly the same, which is below the expectation.
I collected the nsys profile reports as follows:
DeepEP(with normal kernels) enabled:
![Image](https://github.com/user-attachments/assets/0eb030f9-adf2-40a2-b114-f2a926fd52ed)
DeepEP disabled:
![Image](https://github.com/user-attachments/assets/18a6e8c7-7642-40cb-ba1a-0a9f8c1d1e63)
There are no PPs so the SendRecv kernels represent the alltoall kernels.
It seems the combining alltoall comms are faster(13ms->5ms) with DeepEP, but dispatching becomes slower(13ms->16~18ms), and notify_dispatch/cached_notify costs roughly the same time as combining, which makes the e2e throughput not improve.
Any suggestions or insights are appreciated. Thanks!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.