kvcache-ai / kvcache-ai/ktransformers
GGUF format is not working with Deepseek-R1, GLM-4.5-AIR and GPT-OSS
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
Hi, I'm very interested in the GGUF loading with `KT + SGLang`, and successfully ran the Qwen3-30B-A3B demo you provided.
https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel#option-b-llamafile-backend-gguf
I tried to loading GGUF Q4_K_M model generated by [AutoRound](https://github.com/intel/auto-round) and met some issues. BTW, those GGUF models are verified with llama.cpp.
- When I try it with Deepseek R1, it worked well during loading but failed during capturing batches.
- when I try it with GPT-OSS, it reports error during loading.
To rule out AutoRound involvement, I tried gpt-oss-20b-Q4_K_M.gguf from https://huggingface.co/unsloth/gpt-oss-20b-GGUF/tree/main as well, and it still didn't work.
- When it comes to GLM-4.5-AIR, Another error happened as below.
### Reproduction
```bash
# AutoRound quantization
pip install auto-round
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name gpt-oss-120b
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name GLM-4.5-Air
CUDA_VISIBLE_DEVICES=0 auto-round --format gguf:q4_k_m --iters 0 --model_name DeepSeek-R1-0528
# KT + SGLang
## DeepSeek-R1
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
--model DeepSeek-R1-0528 \
--port 8802 \
--kt-method LLAMAFILE \
--kt-weight-path DeepSeek-R1-256x20B-0528-Q4_K_M.gguf \
--kt-cpuinfer 48 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2
## GPT-OSS
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir /mnt/data/models/Qwen3-30B-A3B
### Use AutoRound generated model or this one
huggingface-cli download unsloth/gpt-oss-20b-GGUF gpt-oss-20b-Q4_K_M.gguf \
--local-dir gpt-oss-20b-GGUF
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8801 \
--model gpt-oss-120b \
--trust-remote-code \
--mem-fraction-static 0.92 \
--chunked-prefill-size 4096 \
--served-model-name gpt \
--enable-mixed-chunk \
--kt-method LLAMAFILE \
--kt-weight-path gpt-oss-20b-GGUF \
--kt-cpuinfer 64 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2
## GLM-4.5-AIR
CUDA_VISIBLE_DEVICES=4 \
python -m sglang.launch_server \
--model GLM-4.5-Air \
--port 8802 \
--kt-method LLAMAFILE \
--kt-weight-path GLM-4.5-Air-128x9.4B-Q4_K_M.gguf \
--kt-cpuinfer 64 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 32 \
--kt-max-deferred-experts-per-token 2
```
### Others
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.