NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Integrate SonicMoE kernels
Open
@Butterfingrz is already working on this.
Since Jul 30, 2026.
enhancement
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 141
Description
https://github.com/Dao-AILab/sonic-moe/
-
The current API can only be used without EP, so integrate it for non EP paths.
-
Investigate whether a grouped gemm from sonicmoe could be used with EP. The grouped gemm function would need to implement the following for best perf:
- gemm_gated followed by router weight multiplication in the down projection gemm epilogue similar to gemm_dgated
- Custom backward for the two gemms
Constraints:
Tokens are already permuted for EP so only need the grouped gemm portion from sonic moe, but that API is not yet available.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.