kvcache-ai / kvcache-ai/ktransformers
Can not run on AMD CPU
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 256
On-line CPU(s) list: 0-255
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7713 64-Core Processor
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 64
Socket(s): 2
Stepping: 1
Frequency boost: enabled
CPU max MHz: 2000.0000
CPU min MHz: 1500.0000
BogoMIPS: 3992.63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_goo
d nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_l
m cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cd
p_l3 invpcid_single hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsav
eopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean f
lushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 4 MiB (128 instances)
L1i: 4 MiB (128 instances)
L2: 64 MiB (128 instances)
L3: 512 MiB (16 instances)
NUMA:
NUMA node(s): 8
NUMA node0 CPU(s): 0-15,128-143
NUMA node1 CPU(s): 16-31,144-159
NUMA node2 CPU(s): 32-47,160-175
NUMA node3 CPU(s): 48-63,176-191
NUMA node4 CPU(s): 64-79,192-207
NUMA node5 CPU(s): 80-95,208-223
NUMA node6 CPU(s): 96-111,224-239
NUMA node7 CPU(s): 112-127,240-255
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Spec store bypass: Vulnerable
Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
[sglang] 0:[tmux]*Z "aiplatform-wlf1-ge47-" 22:12 31-Mar-26
### Reproduction
```text
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8000 \
--model /data/phd/hf_models/Qwen3.5-122B-A10B-FP8 \
--kt-weight-path /data/phd/hf_models/Qwen3.5-122B-A10B-FP8 \
--kt-cpuinfer 30 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 1 \
--kt-method BF16 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.98 \
--chunked-prefill-size 4096 \
--max-running-requests 32 \
--max-total-tokens 32000 \
--served-model-name Qwen3.5-35B-A3B \
--enable-mixed-chunk \
--tensor-parallel-size 4 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--disable-custom-all-reduce
```
### Others
error log:
self.model = language_model_cls(
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/models/qwen3_5.py", line 841, in __init__
super().__init__(config=config, quant_config=quant_config, prefix=prefix)
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/models/qwen3_5.py", line 698, in __init__
self.layers = make_layers(
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/utils/common.py", line 648, in make_layers
+ get_offloader().wrap_modules(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/utils/offloader.py", line 36, in wrap_modules
return list(all_modules_generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/utils/common.py", line 650, in
layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/models/qwen3_5.py", line 690, in get_layer
return layer_class(
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/models/qwen3_5.py", line 314, in __init__
self.mlp = Qwen2MoeSparseMoeBlock(
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/models/qwen2_moe.py", line 170, in __init__
self.experts = get_moe_impl_class(quant_config)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 283, in __init__
self.quant_method.create_weights(
File "/opt/conda/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2095, in create_weights
self.wrapper = KTMoEWrapper(
^^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/kt_kernel/experts.py", line 106, in __new__
return backend_cls(
^^^^^^^^^^^^
File "/opt/conda/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 362, in __init__
raise RuntimeError(
RuntimeError: BF16 backend not available. Required ISA:
- AVX512F + AVX512BW + AVX512_BF16
Please recompile kt_kernel_ext with AVX512 + BF16 enabled.
Contributor guide
Research direction
Start by reproducing the command in the issue and reading kt_kernel/utils/amx.py, especially the BF16 backend initialization that raises the error. Trace the call through sglang/srt/layers/moe/kt_ep_wrapper.py and the Qwen model files named in the traceback. Done should include a confirmed AMD CPU compatibility path or a clearly documented supported-ISA limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100