kvcache-ai / kvcache-ai/ktransformers
Capture cuda graph failed: Assertion error when Running DeepSeek V3.2 with SGLang and KT-Kernel
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
CPU: INTEL(R) XEON(R) GOLD 6530 x2
GPU: NVIDIA RTX PRO 4500 Blackwell 32GB [ GB203GL ] x2
RAM: 1TB DDR5
KTransformers CLI v0.1.0
Python 3.11.14
平台 Linux 6.12.63+deb13-amd64
CUDA 13.1
Packages:
kt-kernel 0.5.2.post1
sglang-kt 0.5.2.post2 (sglang-kt)
installed from source .
### Reproduction
I followed the tutorial and ran DeepSeek V3.2 with SGLang and KT-Kernel, but it prompted the following error; how should I adjust it? Additionally, can I run the DeepSeek V3.2 gguf of unsloth with the current kt-kernel?
```text
[2026-03-13 17:36:44 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=13.54 GB
[2026-03-13 17:36:44 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=13.51 GB
[2026-03-13 17:36:44 TP0] Capture cuda graph bs [1, 2, 4, 8, 12, 16]
Capturing batches (bs=16 avail_mem=13.46 GB): 0%| | 0/6 [00:00 N=2112, K=7168, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run `python3 -m sglang.compile_deep_gemm`.
[2026-03-13 17:36:45 TP0] Required memory for warmup: 0.18792724609375GB, Available memory: 13.4566650390625GB
DeepGEMM warmup: 0%| | 0/16384 [00:00, res.text=''
Killed
```
### Others
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 31666 \
--model /nas/DeepSeek-V3.2 \
--kt-weight-path /models/deepseek-v3.2-INT4 \
--kt-cpuinfer 30 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 4 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.7 \
--chunked-prefill-size 2048 \
--max-running-requests 16 \
--max-total-tokens 8192 \
--served-model-name DeepSeek-V3.2 \
--enable-mixed-chunk \
--tensor-parallel-size 2 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--kt-method AMXINT4
python -m sglang.launch_server \
--host 0.0.0.0 --port 31666 \
--model /models/MiniMax-M2.1 \
--kt-weight-path /models/MiniMax-M2.1 \
--kt-cpuinfer 60 --kt-threadpool-count 2 \
--kt-num-gpu-experts 24 --kt-method FP8 \
--kt-gpu-prefill-token-threshold 4096 \
--attention-backend flashinfer --trust-remote-code \
--mem-fraction-static 0.8 --chunked-prefill-size 8192 \
--max-running-requests 16 --max-total-tokens 20000 \
--enable-mixed-chunk --tensor-parallel-size 2 \
--enable-p2p-check --served-model-name MiniMax-M2.1 \
--disable-shared-experts-fusion --fp8-gemm-backend triton
Contributor guide
Research direction
Start by reproducing the DeepSeek V3.2 launch with the reported Python, PyTorch, SGLang and kt-kernel versions, then inspect the CUDA graph path in sglang/srt/model_executor/cuda_graph_runner.py and the DeepGEMM call sites shown in the traceback. Compare the failing GPU and batch configuration with supported recipes; done means identifying a reproducible compatibility fix or documenting the supported configuration, including the MiniMax-M2.1 performance question.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100