kvcache-ai / kvcache-ai/ktransformers

[Bug] run qwen3-235B error with ktransformers:v0.3.2-AVX512 docker image

Open
#1,499 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

I'm having an issue running the qwen3-235b model with the ktransformers:v0.3.2-AVX512 docker image. My local CPU supports the AMX instruction set, and I'm using the Qwen3Moe-serve-amx.yaml as the optimization configuration file. report error:
```
Injecting model.layers.92.self_attn.k_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.92.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbedding
Injecting model.layers.92.mlp as ktransformers.operators.experts . KQwen3MoeSparseMoeBlockV2
Injecting model.layers.92.mlp.gate as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.92.mlp.experts as ktransformers.operators.experts . KTransformersExpertsV2
Injecting model.layers.92.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.92.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.93 as default
Process SpawnProcess-1:
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 181, in __init__
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, weights_loader, device=default_device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 174, in load_weights
load_weights(child, gguf_loader, prefix+name+".", device=device)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/util/utils.py", line 176, in load_weights
module.load()
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1268, in load
self.generate_experts.load(w, warmup=warmup)
File "/opt/conda/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 207, in load
from cpuinfer_ext.moe import AMX_MOEConfig, AMXBF16_MOE
ImportError: cannot import name 'AMX_MOEConfig' from 'cpuinfer_ext.moe' (unknown location)
Injecting model.layers.93.self_attn as ktransformers.operators.balance_serve_attention . KQwen3MoeAttention
Injecting model.layers.93.self_attn.q_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.k_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.v_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.93.self_attn.q_norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
```

### Reproduction

python
- ktransformers/server/main.py
- --model_path=/mnt/models
- --gguf_path=/mnt/models
- --port=8080
- --chunk_size=256
- --cache_lens=32768
- --temperature=0.6
- --top_p=0.95
- --force_think
- --max_new_tokens=10240
- --optimize_config_path=ktransformers/optimize/optimize_rules/Qwen3Moe-serve-amx.yaml
- --max_batch_size=3
- --backend_type=balance_serve
- --model_name=Qwen3-A22-235B
- --cpu_infer=62

### Environment

OS:UOS Server 20
GPU:1 * NVIDIA GeForce RTX 4090 D
CPU:INTEL(R) XEON(R) PLATINUM 8558
Docker image:ktransformers:v0.3.2-AVX512

Contributor guide

Open the contributing guide

Research direction

Start with ktransformers/operators/experts.py at the AMX_MOEConfig import and trace how Qwen3Moe-serve-amx.yaml selects that path. Compare the cpuinfer_ext.moe contents in the v0.3.2-AVX512 image with the operator's expectations, then rerun the supplied server command. Done means the Qwen3-A22-235B model loads without the ImportError.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
ai-infra-agents, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.