ByteDance-Seed / ByteDance-Seed/Triton-distributed

Overlap is slower than non-overlap in MoE+RS on A100*8

Open
#125 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.5k
Forks
172
PR merge metrics
No merged PRs in 30d

Description

I ran the `python/triton_dist/kernels/nvidia/moe_reduce_rs.py`, I find when K is large, overlap version is slower than non-overlap version.
  | M | N | K | num_experts | topk | Triton-Dist(ms) | Triton-non-overlap(ms) | SpeedUp
-- | -- | -- | -- | -- | -- | -- | -- | --
Qwen1.5-MoE-A2.7B | 8192 | 2048 | 1408 | 64 | 4 | 0.592501 | 0.660305 | 1.1143
DeepSeek-MoE | 8192 | 2048 | 1408 | 64 | 6 | 0.827346 | 0.887809 | 1.0730
Mixtral-8x7B | 8192 | 4096 | 14336 | 8 | 2 | 2.259500 | 1.775477 | 0.7858
  | 8192 | 4096 | 1408 | 8 | 2 | 0.679124 | 0.789747 | 1.1629
  | 8192 | 1024 | 14336 | 8 | 2 | 0.676538 | 0.634173 | 0.9372

Below is Qwen1.5-MoE-A2.7B
Image

Below is Mixtral-8x7B
Image

What's the main reason for this phenomenon?

Contributor guide

Open the contributing guide

Research direction

Start by reading python/triton_dist/kernels/nvidia/moe_reduce_rs.py and reproduce the A100*8 comparisons using the M, N, K, num_experts, and topk values listed. Compare the overlap and non-overlap timings for the Qwen, DeepSeek, and Mixtral cases; done means identifying and documenting the main reason the overlap path regresses for large K.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.