kvcache-ai / kvcache-ai/ktransformers

Qwen 3.5 397B-A17B repetitive/looping output when running in bf16+amxint8

Open
#1,880 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

2x 5th-gen Xeon Platinum 64c 128t with 768gb
1x RTX Pro 6000 Workstation Edition 96gb
ktransformers https://github.com/kvcache-ai/ktransformers/commit/7a4b9b0e87467ef1a23c5dd1690d92ca30ca3e26
https://github.com/kvcache-ai/sglang/commit/f6adb4f473ba9a767cd60237ef8325cfcd97eba9

### Reproduction

`kt-kernel` CPU weight conversion command:
```text
python scripts/convert_cpu_weights.py \
--input-path /mnt/data2/models/Qwen3.5-397B-A17B \
--input-type bf16 \
--output /mnt/data2/models/AMX/Qwen3.5-397B-A17B-CPU-NUMA4-AMXINT8 \
--quant-method int8 \
--cpuinfer-threads 128 \
--threadpool-count 4 \
--no-merge-safetensor
```

`sglang+kt-kernel` launch command:
```text
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 5000 \
--model /mnt/data2/models/Qwen3.5-397B-A17B \
--kt-method AMXINT8 \
--kt-weight-path /mnt/data2/models/AMX/Qwen3.5-397B-A17B-CPU-NUMA4-AMXINT8 \
--kt-cpuinfer 128 \
--kt-threadpool-count 4 \
--kt-num-gpu-experts 0 \
--kt-max-deferred-experts-per-token 0 \
--kt-expert-placement-strategy uniform \
--trust-remote-code \
--mem-fraction-static 0.98 \
--served-model-name Qwen/Qwen3.5-397B-A17B \
--enable-mixed-chunk \
--tensor-parallel-size 1 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--disable-custom-all-reduce \
--chunked-prefill-size 4096 \
--context-length 131072 \
--max-total-tokens 131072 \
--max-running-requests 1 \
--attention-backend triton \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
```

Attempting to generate results in repetitive output:
Image
(opencode, default sampling settings from `generation_config.json`)

### Others

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with scripts/convert_cpu_weights.py and the documented sglang.launch_server command, using the listed bf16 and AMXINT8 settings. Compare the generated output with the repetitive result shown in the report and trace the kt-kernel integration involved in CPU weight conversion and inference. Done means identifying the cause and demonstrating non-repetitive generation under the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.