kvcache-ai / kvcache-ai/ktransformers
Qwen 3.5 397B-A17B repetitive/looping output when running in bf16+amxint8
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
2x 5th-gen Xeon Platinum 64c 128t with 768gb
1x RTX Pro 6000 Workstation Edition 96gb
ktransformers https://github.com/kvcache-ai/ktransformers/commit/7a4b9b0e87467ef1a23c5dd1690d92ca30ca3e26
https://github.com/kvcache-ai/sglang/commit/f6adb4f473ba9a767cd60237ef8325cfcd97eba9
### Reproduction
`kt-kernel` CPU weight conversion command:
```text
python scripts/convert_cpu_weights.py \
--input-path /mnt/data2/models/Qwen3.5-397B-A17B \
--input-type bf16 \
--output /mnt/data2/models/AMX/Qwen3.5-397B-A17B-CPU-NUMA4-AMXINT8 \
--quant-method int8 \
--cpuinfer-threads 128 \
--threadpool-count 4 \
--no-merge-safetensor
```
`sglang+kt-kernel` launch command:
```text
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 5000 \
--model /mnt/data2/models/Qwen3.5-397B-A17B \
--kt-method AMXINT8 \
--kt-weight-path /mnt/data2/models/AMX/Qwen3.5-397B-A17B-CPU-NUMA4-AMXINT8 \
--kt-cpuinfer 128 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 0 \
--kt-max-deferred-experts-per-token 0 \
--kt-expert-placement-strategy uniform \
--trust-remote-code \
--mem-fraction-static 0.98 \
--served-model-name Qwen/Qwen3.5-397B-A17B \
--enable-mixed-chunk \
--tensor-parallel-size 1 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--disable-custom-all-reduce \
--chunked-prefill-size 4096 \
--context-length 131072 \
--max-total-tokens 131072 \
--max-running-requests 1 \
--attention-backend triton \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
```
Attempting to generate results in repetitive output:
(opencode, default sampling settings from `generation_config.json`)
### Others
_No response_
Contributor guide
Research direction
Start by reproducing the issue with scripts/convert_cpu_weights.py and the documented sglang.launch_server command, using the listed bf16 and AMXINT8 settings. Compare the generated output with the repetitive result shown in the report and trace the kt-kernel integration involved in CPU weight conversion and inference. Done means identifying the cause and demonstrating non-repetitive generation under the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100