deepseek-ai / deepseek-ai/DeepGEMM

[Bug] k_grouped_fp8_gemm_nt_contiguous crashes with n = 768 on H100

Open
#237 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
7.8k
Forks
1.3k
Avg merge
3d 7h
Merged PRs (30d)
3

Description

First of all, thank you for the amazing work on DeepGEMM — it's been extremely helpful.
While integrating DeepGEMM into a backward pass implementation, I encountered a reproducible crash when running the **k-grouped FP8 GEMM** with **N = 768**.

---

## **❗ Error**

Running DeepGEMM with the following shape causes a CUDA illegal instruction error:

```
RuntimeError: CUDA driver error (csrc/apis/../jit_kernels/impls/sm90_fp8_gemm_1d1d.hpp:65): 715
(CUDA_ERROR_ILLEGAL_INSTRUCTION, an illegal instruction was encountered)
```

---

## **🔁 Reproduction**

The issue reproduces consistently by adding the following shape into
`enumerate_k_grouped_contiguous`:

```python
(128, 2048, 768, 4096)
```

This triggers the following call path:

* `k_grouped_fp8_gemm_nt_contiguous`
* → FP8 kernel selection
* → SM90 kernel dispatch
* → crash with CUDA illegal instruction

Notably, the same configuration works correctly when **N = 1536**, so the issue appears to be specific to `N = 768`.

---

## **🧩 Expected Behavior**

The kernel should run successfully for `(groups=128, M=2048, N=768, K=4096)` without causing an illegal instruction.

---

## **🧪 Environment (if helpful)**

* GPU: H100 (SM90)
* CUDA Toolkit: CUDA 12.9 Driver Version: 535.161.08
* PyTorch version: 2.8.0

---

## **🙏 Additional Notes**

If you need further logs or want me to test a patch, I’m happy to help.

Thanks again for the excellent work on DeepGEMM!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.