kvcache-ai / kvcache-ai/ktransformers
GLM5.2 int8: 'AMXMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer'
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
System: GPU: 4xA100/80 CPU: 2xAMD EPYC 9254, Platform: Linux 7.0.11-arch1-1
KMD Version: 610.43.02 CUDA UMD Version: 13.3
Python: 3.11 in conda venv:
CUDA 12.8
Packages:
kt-kernel 0.6.4
sglang-kt 0.6.3.post1 (sglang-kt)
### Reproduction
- prepare conda venv (install compilers, libraries matching CUDA 12.8, nvcc for 12.8, compile ktransformers, set env vars to compile with venv compilers and link w venv libraries)
- download GLM 5.2 FP16
- convert CPU weights:
`python convert_cpu_weights.py --input-path '/home/user/LLMs/weights/GLM-5.2-FP16' --input-type fp16 --output '/home/user/LLMs/weights/GLM-5.2-int8-CPU' --quant-method int8 --cpuinfer-threads 48 --threadpool-count 8 --no-merge-safetensor `
- use original BF16 GPU weights (because conversion fails, but it is different topic)
- launch:
```
python -m sglang.launch_server \
--model-path '/home/user/LLMs/weights/GLM-5.2-FP16' \
--kt-weight-path '/home/user/LLMs/weights/GLM-5.2-int8-CPU' \
--kt-cpuinfer 48 \
--kt-threadpool-count 8 \
--kt-num-gpu-experts 30 \
--kt-method AMXINT8 \
--kt-gpu-prefill-token-threshold 512 \
--kt-enable-dynamic-expert-update \
--kt-expert-placement-strategy uniform \
--kt-max-deferred-experts-per-token 1 \
--chunked-prefill-size 16384 \
--enable-p2p-check \
--enable-mixed-chunk \
--sleep-on-idle \
--watchdog-timeout 3000 \
--tp-size 4 \
--cuda-graph-max-bs 4 \
--trust-remote-code \
--mem-fraction-static 0.95 \
--kv-cache-dtype auto \
--max-total-tokens 192000 \
--max-running-requests 4 \
--attention-backend triton \
--fp8-gemm-backend auto \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--served-model-name GLM5.2i \
--host 127.0.0.1 \
--port 1027 --dtype auto
```
-result: model running
-send a request
-result: crash + long traceback (attached)
[log.log](https://github.com/user-attachments/files/30427310/log.log)
### Others
flashinfer is 0.6.3
GLM5.2 w BF16 - works ~ 6.6 tk/s
Contributor guide
Research direction
Start with the attached log.log and the AMXMoEWrapper attribute error, then reproduce using the provided sglang.launch_server command with --kt-method AMXINT8. Check the AMXINT8 path after a request is sent and determine why submit_write_weight_scale_to_buffer is unavailable. Done means the request completes without this crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100