kvcache-ai / kvcache-ai/ktransformers

GGUF format is not working with Deepseek-R1, GLM-4.5-AIR and GPT-OSS

Open
#1,655 2 comments 0 reactions 1 assignee Claimed by @SkqLiao View on GitHub
bug
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

Hi, I'm very interested in the GGUF loading with `KT + SGLang`, and successfully ran the Qwen3-30B-A3B demo you provided.
https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel#option-b-llamafile-backend-gguf

I tried to loading GGUF Q4_K_M model generated by [AutoRound](https://github.com/intel/auto-round) and met some issues. BTW, those GGUF models are verified with llama.cpp.

- When I try it with Deepseek R1, it worked well during loading but failed during capturing batches.

Image

Image

Image

- when I try it with GPT-OSS, it reports error during loading.
To rule out AutoRound involvement, I tried gpt-oss-20b-Q4_K_M.gguf from https://huggingface.co/unsloth/gpt-oss-20b-GGUF/tree/main as well, and it still didn't work.
Image

- When it comes to GLM-4.5-AIR, Another error happened as below.
Image

### Reproduction

```bash
# AutoRound quantization
pip install auto-round
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name gpt-oss-120b
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name GLM-4.5-Air
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name DeepSeek-R1-0528

# KT + SGLang
## DeepSeek-R1
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
--model DeepSeek-R1-0528 \
--port 8802 \
--kt-method LLAMAFILE \
--kt-weight-path DeepSeek-R1-256x20B-0528-Q4_K_M.gguf \
--kt-cpuinfer 48 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2

## GPT-OSS
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B
### Use AutoRound generated model or this one
huggingface-cli download unsloth/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \
--local-dir gpt-oss-20b-GGUF

python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8801 \
--model gpt-oss-120b \
--trust-remote-code \
--mem-fraction-static 0.92 \
--chunked-prefill-size 4096 \
--served-model-name gpt \
--enable-mixed-chunk \
--kt-method LLAMAFILE \
--kt-weight-path gpt-oss-20b-GGUF \
--kt-cpuinfer 64 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2

## GLM-4.5-AIR
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
--model GLM-4.5-Air \
--port 8802 \
--kt-method LLAMAFILE \
--kt-weight-path GLM-4.5-Air-128x9.4B-Q4_K_M.gguf \
--kt-cpuinfer 64 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2

```

### Others

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.