kvcache-ai / kvcache-ai/ktransformers

[Bug] 运行DeepSeek-R1-Q8,在backend为AMXInt8下,出现coredump

Open
#1,392 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

在测试DeepSeek-R1-Q8模型时,backend为AMXInt8下,运行服务在加载模型到第6层时出现coredump
[root@db614865551d ktransformers]# vi /usr/local/lib64/python3.12/site-packages/ktransformers/operators/experts.py
[root@db614865551d ktransformers]# ktransformers --model_path /data/ssdmodels/DeepSeek-R1-Q4_K_M/ --gguf_path /data/ssdmodels/deepseekr1-q8/ --model_name DeepSeek-R1-Q8 --host 0.0.0.0 --port 8899 --cpu_infer 95 --max_new_tokens 8192 --optimize_config_path /usr/local/lib64/python3.12/site-packages/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yaml --cache_lens 32768 --batch_size 1
2025-06-17 02:15:16,264 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
Injecting model.layers.1 as default
Injecting model.layers.1.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.1.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.q_a_layernorm as default
Injecting model.layers.1.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.kv_a_layernorm as default
Injecting model.layers.1.self_attn.kv_b_proj as default
Injecting model.layers.1.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.1.mlp as default
Injecting model.layers.1.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.1.mlp.act_fn as default
Injecting model.layers.1.input_layernorm as default
Injecting model.layers.1.post_attention_layernorm as default
Injecting model.layers.2 as default
Injecting model.layers.2.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.2.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.q_a_layernorm as default
Injecting model.layers.2.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.kv_a_layernorm as default
Injecting model.layers.2.self_attn.kv_b_proj as default
Injecting model.layers.2.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.2.mlp as default
Injecting model.layers.2.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.2.mlp.act_fn as default
Injecting model.layers.2.input_layernorm as default
Injecting model.layers.2.post_attention_layernorm as default
Injecting model.layers.3 as default
Injecting model.layers.3.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.3.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.q_a_layernorm as default
Injecting model.layers.3.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.kv_a_layernorm as default
Injecting model.layers.3.self_attn.kv_b_proj as default
Injecting model.layers.3.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.3.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.3.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.3.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.3.mlp.shared_experts as default
Injecting model.layers.3.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.3.mlp.shared_experts.act_fn as default
Injecting model.layers.3.input_layernorm as default
Injecting model.layers.3.post_attention_layernorm as default
Injecting model.layers.4 as default
Injecting model.layers.4.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.4.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.q_a_layernorm as default
Injecting model.layers.4.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.kv_a_layernorm as default
Injecting model.layers.4.self_attn.kv_b_proj as default
Injecting model.layers.4.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.4.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.4.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.4.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.4.mlp.shared_experts as default
Injecting model.layers.4.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.4.mlp.shared_experts.act_fn as default
Injecting model.layers.4.input_layernorm as default
Injecting model.layers.4.post_attention_layernorm as default
Injecting model.layers.5 as default
Injecting model.layers.5.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.5.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.q_a_layernorm as default
Injecting model.layers.5.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.kv_a_layernorm as default
Injecting model.layers.5.self_attn.kv_b_proj as default
Injecting model.layers.5.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.5.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.5.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.5.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.5.mlp.shared_experts as default
Injecting model.layers.5.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.5.mlp.shared_experts.act_fn as default
Injecting model.layers.5.input_layernorm as default
Injecting model.layers.5.post_attention_layernorm as default
Injecting model.layers.6 as default
Injecting model.layers.6.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.6.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.q_a_layernorm as default
Injecting model.layers.6.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.kv_a_layernorm as default
Injecting model.layers.6.self_attn.kv_b_proj as default
Injecting model.layers.6.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.6.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.6.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.6.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.6.mlp.shared_experts as default
Injecting model.layers.6.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.6.mlp.shared_experts.act_fn as default
Injecting model.layers.6.input_layernorm as default
Injecting model.layers.6.post_attention_layernorm as default
Injecting model.layers.7 as default
Injecting model.layers.7.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.7.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.q_a_layernorm as default
Injecting model.layers.7.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.kv_a_layernorm as default
Injecting model.layers.7.self_attn.kv_b_proj as default
Injecting model.layers.7.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.7.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.7.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.7.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.7.mlp.shared_experts as default
Injecting model.layers.7.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.7.mlp.shared_experts.act_fn as default
Injecting model.layers.7.input_layernorm as default
Injecting model.layers.7.post_attention_layernorm as default
Injecting model.layers.8 as default
Injecting model.layers.8.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.8.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.q_a_layernorm as default
Injecting model.layers.8.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.kv_a_layernorm as default
Injecting model.layers.8.self_attn.kv_b_proj as default
Injecting model.layers.8.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3
Injecting model.layers.8.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.8.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.8.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.8.mlp.shared_experts as default
Injecting model.layers.8.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.8.mlp.shared_experts.act_fn as default
Injecting model.layers.8.input_layernorm as default
Injecting model.layers.8.post_attention_layernorm as default
Injecting model.layers.9 as default
..................
loading model.layers.1.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.1.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.1.input_layernorm.weight to cuda:0
loading model.layers.1.post_attention_layernorm.weight to cuda:0
loading model.layers.2.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.2.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.2.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.2.input_layernorm.weight to cuda:0
loading model.layers.2.post_attention_layernorm.weight to cuda:0
loading model.layers.3.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.3.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.3.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.3.input_layernorm.weight to cuda:0
loading model.layers.3.post_attention_layernorm.weight to cuda:0
loading model.layers.4.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.4.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.4.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.4.input_layernorm.weight to cuda:0
loading model.layers.4.post_attention_layernorm.weight to cuda:0
loading model.layers.5.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.5.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.5.self_attn.kv_b_proj.weight to cuda:0
loading model.layers.5.input_layernorm.weight to cuda:0
loading model.layers.5.post_attention_layernorm.weight to cuda:0
loading model.layers.6.self_attn.q_a_layernorm.weight to cuda:0
loading model.layers.6.self_attn.kv_a_layernorm.weight to cuda:0
loading model.layers.6.self_attn.kv_b_proj.weight to cuda:0
Segmentation fault

### Reproduction

ktransformers --model_path /data/ssdmodels/DeepSeek-R1-Q4_K_M/ --gguf_path /data/ssdmodels/deepseekr1-q8/ --model_name DeepSeek-R1-Q8 --host 0.0.0.0 --port 8899 --cpu_infer 95 --max_new_tokens 8192 --optimize_config_path /usr/local/lib64/python3.12/site-packages/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yaml --cache_lens 32768 --batch_size 1

### Environment

系统:欧拉8
内核:5.10.134
cuda:12.4
torch:2.6.0+cu124
glibc: 2.38-19
transformers: 4.43.2
ktransformers: main分支(32f3d7befb906ddc2563e43dda1ad04703272f49)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported ktransformers command with the DeepSeek-R1-Q8 model and AMXInt8 configuration, focusing on loading around model layer 6. Inspect ktransformers/operators/experts.py and the referenced DeepSeek-V3-Chat-amx.yaml configuration. Done means identifying the coredump cause and confirming that this configuration loads without a coredump.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.