kvcache-ai / kvcache-ai/ktransformers

Capture cuda graph failed: Assertion error when Running DeepSeek V3.2 with SGLang and KT-Kernel

Open
#1,885 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

CPU: INTEL(R) XEON(R) GOLD 6530 x2
GPU: NVIDIA RTX PRO 4500 Blackwell 32GB [ GB203GL ] x2
RAM: 1TB DDR5

KTransformers CLI v0.1.0

Python 3.11.14
平台 Linux 6.12.63+deb13-amd64
CUDA 13.1

Packages:

kt-kernel 0.5.2.post1
sglang-kt 0.5.2.post2 (sglang-kt)

installed from source .

### Reproduction

I followed the tutorial and ran DeepSeek V3.2 with SGLang and KT-Kernel, but it prompted the following error; how should I adjust it? Additionally, can I run the DeepSeek V3.2 gguf of unsloth with the current kt-kernel?

```text
[2026-03-13 17:36:44 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=13.54 GB
[2026-03-13 17:36:44 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=13.51 GB
[2026-03-13 17:36:44 TP0] Capture cuda graph bs [1, 2, 4, 8, 12, 16]
Capturing batches (bs=16 avail_mem=13.46 GB): 0%| | 0/6 [00:00 N=2112, K=7168, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run `python3 -m sglang.compile_deep_gemm`.
[2026-03-13 17:36:45 TP0] Required memory for warmup: 0.18792724609375GB, Available memory: 13.4566650390625GB
DeepGEMM warmup: 0%| | 0/16384 [00:00, res.text=''

Killed
```

### Others

python -m sglang.launch_server \
--host 0.0.0.0 \
--port 31666 \
--model /nas/DeepSeek-V3.2 \
--kt-weight-path /models/deepseek-v3.2-INT4 \
--kt-cpuinfer 30 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 4 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.7 \
--chunked-prefill-size 2048 \
--max-running-requests 16 \
--max-total-tokens 8192 \
--served-model-name DeepSeek-V3.2 \
--enable-mixed-chunk \
--tensor-parallel-size 2 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--kt-method AMXINT4

python -m sglang.launch_server \
--host 0.0.0.0 --port 31666 \
--model /models/MiniMax-M2.1 \
--kt-weight-path /models/MiniMax-M2.1 \
--kt-cpuinfer 60 --kt-threadpool-count 2 \
--kt-num-gpu-experts 24 --kt-method FP8 \
--kt-gpu-prefill-token-threshold 4096 \
--attention-backend flashinfer --trust-remote-code \
--mem-fraction-static 0.8 --chunked-prefill-size 8192 \
--max-running-requests 16 --max-total-tokens 20000 \
--enable-mixed-chunk --tensor-parallel-size 2 \
--enable-p2p-check --served-model-name MiniMax-M2.1 \
--disable-shared-experts-fusion --fp8-gemm-backend triton

Image

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the DeepSeek V3.2 launch with the reported Python, PyTorch, SGLang and kt-kernel versions, then inspect the CUDA graph path in sglang/srt/model_executor/cuda_graph_runner.py and the DeepGEMM call sites shown in the traceback. Compare the failing GPU and batch configuration with supported recipes; done means identifying a reproducible compatibility fix or documenting the supported configuration, including the MiniMax-M2.1 performance question.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.