ByteDance-Seed / ByteDance-Seed/Triton-distributed
Overlap is slower than non-overlap in MoE+RS on A100*8
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 172
- PR merge metrics
- No merged PRs in 30d
Description
I ran the `python/triton_dist/kernels/nvidia/moe_reduce_rs.py`, I find when K is large, overlap version is slower than non-overlap version.
| M | N | K | num_experts | topk | Triton-Dist(ms) | Triton-non-overlap(ms) | SpeedUp
-- | -- | -- | -- | -- | -- | -- | -- | --
Qwen1.5-MoE-A2.7B | 8192 | 2048 | 1408 | 64 | 4 | 0.592501 | 0.660305 | 1.1143
DeepSeek-MoE | 8192 | 2048 | 1408 | 64 | 6 | 0.827346 | 0.887809 | 1.0730
Mixtral-8x7B | 8192 | 4096 | 14336 | 8 | 2 | 2.259500 | 1.775477 | 0.7858
| 8192 | 4096 | 1408 | 8 | 2 | 0.679124 | 0.789747 | 1.1629
| 8192 | 1024 | 14336 | 8 | 2 | 0.676538 | 0.634173 | 0.9372
Below is Qwen1.5-MoE-A2.7B
Below is Mixtral-8x7B
What's the main reason for this phenomenon?
Contributor guide
Research direction
Start by reading python/triton_dist/kernels/nvidia/moe_reduce_rs.py and reproduce the A100*8 comparisons using the M, N, K, num_experts, and topk values listed. Compare the overlap and non-overlap timings for the Qwen, DeepSeek, and Mixtral cases; done means identifying and documenting the main reason the overlap path regresses for large K.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100