deepseek-ai / deepseek-ai/DeepEP
Question about launch configs of normal kernels
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
Hi~first of all, thank you for the amazing work !
I have some questions about the necessity of the launch configs for normal kernels.
I notice that for both `intranode` and `internode` normal kernels, you uses the same macro `SETUP_LAUNCH_CONFIG` to set the `cudaLaunchAttribute` with two attributes that **affect computation and communication overlapping**:
1. `cudaLaunchAttributeCooperative` (`cooperative = 1`), which enforces all the blocks to be launched simultaneously, resulting in **the SM margin value left by the computation (*persistent*) kernels to be at least as large as the number of SMs used by comm kernels (`num_sms`)**, with the constraint "one block per SM" mentioned in the previous [issue](https://github.com/deepseek-ai/DeepEP/issues/44).
2. `cudaLaunchAttributeClusterDimension` (`clusterDim.x = 2`), which requires each pair of two SMs to be positioned in the same GPC, resulting in **even larger SM margin value than `num_sms`**, since leaving exact `num_sms` SMs still can not guarantee any GPC has left at least two SMs.
However, I've walked through the souce code of normal kernels, but found it a block-level kernel without the usage of any cluster features (e.g. `cg::this_grid().sync` or distributed shared memory), except for the TMA multicast used in `tma_load_1d` (*and if I overlooked something, please point me to it*).
It is generally admitted that SM margin seems too expensive to be set with a large value, especially quite a waste for the timeline range when the overlapped comm kernel is finished but the computation kernel has not.
And from my experience, NCCL only requires a small value of SM margin (`4~8`) to be fully overlapped with the computation kernel (*while NCCL has incorporated CGA to leverage cluster capabilities since version 2.16, setting `NCCL_CGA_CLUSTER_SIZE=1` is acceptable and still delivers excellent performance.*).
Therefore, I wonder the necessity to attach these two kernels with two aggressive attributes above:
1. Are the two attributes kept solely for TMA efficiency? If so, what degradation should we expect when they are removed ?
2. Are there any other reasons to keep these two attributes ? If so, is there any chance to design a good-enough normal kernel that avoids these hard constraints like NCCL ?
Looking forward to your insights !
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.