deepseek-ai / deepseek-ai/DeepGEMM

[Question] Do you have any plans to improve grouped wgrad gemm?

Open
#125 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Cuda
Stars
7.8k
Forks
1.3k
Avg merge
3d 7h
Merged PRs (30d)
3

Description

First of all, thank you for sharing your great implementation.
We are experimenting with using some non-gemm kernels (e.g. quantize, transpose, permute, 128-align per group) with DeepGEMM to train our moe models.
In practice, we observed significant speedups over [TransformerEngine GroupedLinear](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/api/pytorch.html#transformer_engine.pytorch.GroupedLinear) for fprop and dgrad, but negligible performance gains for wgrad.
It seems that there is something that can be optimized, since it launches a non-grouped version of wgrad for each group separately.
TransformerEngine has a similar implementation, but at least it utilizes [multi-stream](https://github.com/NVIDIA/TransformerEngine/blob/f05f12c974b37c5bd8dfca3d2d294be53b66abfa/transformer_engine/common/gemm/cublaslt_gemm.cu#L629).
Do you have any plans or hints to improve these?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.