deepseek-ai / deepseek-ai/DeepEP

Question about launch configs of normal kernels

Open
#526 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hi~first of all, thank you for the amazing work !

I have some questions about the necessity of the launch configs for normal kernels.

I notice that for both `intranode` and `internode` normal kernels, you uses the same macro `SETUP_LAUNCH_CONFIG` to set the `cudaLaunchAttribute` with two attributes that **affect computation and communication overlapping**:
1. `cudaLaunchAttributeCooperative` (`cooperative = 1`), which enforces all the blocks to be launched simultaneously, resulting in **the SM margin value left by the computation (*persistent*) kernels to be at least as large as the number of SMs used by comm kernels (`num_sms`)**, with the constraint "one block per SM" mentioned in the previous [issue](https://github.com/deepseek-ai/DeepEP/issues/44).
2. `cudaLaunchAttributeClusterDimension` (`clusterDim.x = 2`), which requires each pair of two SMs to be positioned in the same GPC, resulting in **even larger SM margin value than `num_sms`**, since leaving exact `num_sms` SMs still can not guarantee any GPC has left at least two SMs.

However, I've walked through the souce code of normal kernels, but found it a block-level kernel without the usage of any cluster features (e.g. `cg::this_grid().sync` or distributed shared memory), except for the TMA multicast used in `tma_load_1d` (*and if I overlooked something, please point me to it*).

It is generally admitted that SM margin seems too expensive to be set with a large value, especially quite a waste for the timeline range when the overlapped comm kernel is finished but the computation kernel has not.

And from my experience, NCCL only requires a small value of SM margin (`4~8`) to be fully overlapped with the computation kernel (*while NCCL has incorporated CGA to leverage cluster capabilities since version 2.16, setting `NCCL_CGA_CLUSTER_SIZE=1` is acceptable and still delivers excellent performance.*).

Therefore, I wonder the necessity to attach these two kernels with two aggressive attributes above:
1. Are the two attributes kept solely for TMA efficiency? If so, what degradation should we expect when they are removed ?
2. Are there any other reasons to keep these two attributes ? If so, is there any chance to design a good-enough normal kernel that avoids these hard constraints like NCCL ?

Looking forward to your insights !

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.