deepseek-ai / deepseek-ai/DeepEP

How to reduce the shared memory footprint of the low latency combine kernel?

Open
#362 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

I am encountering a runtime error when running the `internode_ll::combine kernel` with hidden=2048. The error is `cudaFuncSetAttribute` failed, which is caused by the high shared memory requirement of the kernel's send phase approx. (200KB).

```
RuntimeError: Failed: Assertion error DeepEP/csrc/kernels/internode_ll.cu:985 'cudaFuncSetAttribute(combine_func, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size) == cudaSuccess'
```
I identified three main parameters that control the shared memory size: `hidden`,`kNumStages`,`kNumMaxUnrolls`

To resolve the runtime error, I need to reduce the shared memory allocation. Given that the hidden size is fixed(2048), which of the other two parameters is the intended or safest one to adjust?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.