kvcache-ai / kvcache-ai/ktransformers
[Bug] ktransformers 0.31跑deepseekr1报错assert self.gate_type == GGMLQuantizationType.BF16
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
我使用ktransformers-0.3.1版本跑deepseekr1,CPU是Intel(R) Xeon(R) Platinum 8468V,运行指令是USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yaml,
报错日志如下:
loading model.layers.1.self_attn.q_a_layernorm.weight to cuda
loading model.layers.1.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.1.self_attn.kv_b_proj.weight to cuda
loading model.layers.1.input_layernorm.weight to cuda
loading model.layers.1.post_attention_layernorm.weight to cuda
loading model.layers.2.self_attn.q_a_layernorm.weight to cuda
loading model.layers.2.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.2.self_attn.kv_b_proj.weight to cuda
loading model.layers.2.input_layernorm.weight to cuda
loading model.layers.2.post_attention_layernorm.weight to cuda
loading model.layers.3.self_attn.q_a_layernorm.weight to cuda
loading model.layers.3.self_attn.kv_a_layernorm.weight to cuda
loading model.layers.3.self_attn.kv_b_proj.weight to cuda
Traceback (most recent call last):
File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 191, in
fire.Fire(local_chat)
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/anaconda3/envs/kt/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/kt-0.3/ktransformers/./ktransformers/local_chat.py", line 118, in local_chat
optimize_and_load_gguf(model, optimize_config_path, gguf_path, config, default_device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, weights_loader, device=default_device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 164, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/home/kt-0.3/ktransformers/./ktransformers/util/utils.py", line 166, in load_weights
module.load()
File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 684, in load
self.generate_experts.load(w, warmup=warmup)
File "/home/kt-0.3/ktransformers/./ktransformers/operators/experts.py", line 226, in load
assert self.gate_type == GGMLQuantizationType.BF16
具体配置及报错见下图:
请问下intel的amx优化是只支持BF16的权重在线量化成int8(CPU端)及int4(GPU端)么?是否支持DeepSeek-R1-Q4_K_M?amx具体的优势场景有说明么?
### Reproduction
USE_NUMA=1 python3 ./ktransformers/local_chat.py --model_path /home/DeepSeek-R1-GGUF/ --gguf_path /home/deepseek-r1/ --cpu_infer 65 --max_new_tokens 1000 --optimize-config-path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-amx.yam
DeepSeek-R1-Q4_K_M
### Environment
L20 GPU Intel(R) Xeon(R) Platinum 8468V CPU
Contributor guide
Research direction
Start with ktransformers/local_chat.py and ktransformers/optimize/optimize.py, then trace the loading path through ktransformers/util/utils.py, ktransformers/operators/base_operator.py, and ktransformers/operators/experts.py. Reproduce the stated DeepSeek-R1-Q4_K_M command and determine how the AMX optimization configuration reaches the gate-type assertion; done means the supported model/config combination is established and the incompatibility is fixed or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100