deepseek-ai / deepseek-ai/DeepEP

Can we reduce the kernel latency for normal dispatch and combine when overlapping with gemm kernels through tuning the UNROLL_FACTOR?

Open
#184 6 comments 3 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hello, we have a question about intra-node copy efficiency that influence the performance and we appreciate your help!
We notice degraded performance of communication kernels when overlapping communication and computation. Through analysis, we think that the performance bottleneck shifted from network bandwidth limitations to intra-node copy under overlapping scenarios. To address this, we adjust the UNROLL_FACTOR parameter in intra-node copy function UNROLLED_WARP_COPY to reduce copy latency.
Here's some result (via a simple benchmark test simulating communication-computation overlap):

| Case(num_tokens=8k) \ Latency(ms) | dispatch | combine | performance degradation|
|-------|-------|-------|-------|
| DeepEP | 3.54 | 6.91 | 0% |
| DeepEP+overlap | 4.18 | 6.97 | dispatch: 18.1% combine: 0.87% |
| DeepEP+overlap+adapted UNROLL_FACTOR | 3.82 | 6.94 | dispatch: 7.9% combine: 0.14%|

Single-stream tests: Performance remains largely unchanged
overlap tests: Mitigating the observed performance degradation

This parameter tuning is also effective when the number of SM used by communication is small (guessing that intra-node copy is the bottleneck at this time). For example, when the number of SM is reduced to 12 or 18, adjusting UNROLL_FACTOR can reduce the dispatch latency by 11%.

Therefore, we are very curious about the appropriate setting of this parameter and the reason for setting it this way.
We look forward to your suggestions. Thank you!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.