kvcache-ai / kvcache-ai/ktransformers
[Bug] run qwen3-235B error with ktransformers:v0.3.2-AVX512 docker image
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
I'm having an issue running the qwen3-235b model with the ktransformers:v0.3.2-AVX512 docker image. My local CPU supports the AMX instruction set, and I'm using the Qwen3Moe-serve-amx.yaml as the optimization configuration file. report error:
```
Injecting model.layers.92.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.92.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.92.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2
Injecting model.layers.92.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.92.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.92.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.92.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.93 as default
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 181, in __init__
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, weights_loader, device=default_device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1268, in load
self.generate_experts.load(w, warmup=warmup)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 207, in load
from cpuinfer_ext.moe import AMX_MOEConfig, AMXBF16_MOE
ImportError: cannot import name 'AMX_MOEConfig' from 'cpuinfer_ext.moe' (unknown location)
Injecting model.layers.93.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention
Injecting model.layers.93.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
```
### Reproduction
python
- ktransformers/server/main.py
- --model_path=/mnt/models
- --gguf_path=/mnt/models
- --port=8080
- --chunk_size=256
- --cache_lens=32768
- --temperature=0.6
- --top_p=0.95
- --force_think
- --max_new_tokens=10240
- --optimize_config_path=ktransformers/optimize/optimize_rules/Qwen3Moe-serve-amx.yaml
- --max_batch_size=3
- --backend_type=balance_serve
- --model_name=Qwen3-A22-235B
- --cpu_infer=62
### Environment
OS:UOS Server 20
GPU:1 * NVIDIA GeForce RTX 4090 D
CPU:INTEL(R) XEON(R) PLATINUM 8558
Docker image:ktransformers:v0.3.2-AVX512
Contributor guide
Research direction
Start with ktransformers/operators/experts.py at the AMX_MOEConfig import and trace how Qwen3Moe-serve-amx.yaml selects that path. Compare the cpuinfer_ext.moe contents in the v0.3.2-AVX512 image with the operator's expectations, then rerun the supplied server command. Done means the Qwen3-A22-235B model loads without the ImportError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- ai-infra-agents, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100