deepseek-ai / deepseek-ai/DeepGEMM
[Question] Do you have any plans to improve grouped wgrad gemm?
- Dominant language
- Cuda
- Stars
- 7.8k
- Forks
- 1.3k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 3
Description
First of all, thank you for sharing your great implementation.
We are experimenting with using some non-gemm kernels (e.g. quantize, transpose, permute, 128-align per group) with DeepGEMM to train our moe models.
In practice, we observed significant speedups over [TransformerEngine GroupedLinear](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/api/pytorch.html#transformer_engine.pytorch.GroupedLinear) for fprop and dgrad, but negligible performance gains for wgrad.
It seems that there is something that can be optimized, since it launches a non-grouped version of wgrad for each group separately.
TransformerEngine has a similar implementation, but at least it utilizes [multi-stream](https://github.com/NVIDIA/TransformerEngine/blob/f05f12c974b37c5bd8dfca3d2d294be53b66abfa/transformer_engine/common/gemm/cublaslt_gemm.cu#L629).
Do you have any plans or hints to improve these?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.