kvcache-ai / kvcache-ai/ktransformers
GLM5.2 on RTX 6000 Pro ( sm120 ) * Update
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
Hi,
Just a little update on Kt-kernel FP8 sm120 support. I managed to get it working by having codex backport sm120 support onto the kvcache-ai/sglang branch ( based off of this https://github.com/NVIDIA/nvidia-gcp-samples/pull/25 ). You can find that branch at https://github.com/trilog-inc/sglang/tree/codex/backport-sm120-glm52-kt . There is an accompanying Ktransformers fork https://github.com/trilog-inc/ktransformers/tree/codex/sm120-glm52-install-docs. The changes are mostly on the automated install.sh script to explicitly set the sm120 arch and to point the sglang install to a local directory instead of the kvcache branch.
A few changes to the nsa backend had to be done in order to accommodate the new flashinfer and deep_gemm versions, however the fix is pretty straightforward. The heavy lifting is being done by the updated flashinfer version (post-0.6.13) and the nv_dev branch of DeepGEMM ( nv_dev @ a6b593d ). Make sure to recompile the sgl-kernel once the nv_dev branch of deepGemm is pulled and built.
On a w7-3465x Xeon with 768GB DDR5 @ 5600 ( AMX ) and a single RTX 6000 Pro, I get prompt processing speeds of ~120t/s and generation of about 8t/s. I ran an entire OpenCode session and everything looks good.
If anyone is curious to try, please report performance. The next step would be to work out NVFP4 for increased performance but nominal intelligence drop.
### Reproduction
Launch command:
```text
SGLANG_ENABLE_JIT_DEEPGEMM=1 FLASHINFER_CUDA_ARCH_LIST=12.0f CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server
--model-path /mnt/home_extend/models/GLM52
--kt-weight-path /mnt/home_extend/models/GLM52
--kt-cpuinfer 50
--kt-threadpool-count 2
--kt-num-gpu-experts 20
--kt-method FP8
--kt-gpu-prefill-token-threshold 1024
--kt-enable-dynamic-expert-update
--kt-expert-placement-strategy uniform
--tp-size 1
--trust-remote-code
--host 0.0.0.0
--port 60000
--mem-fraction-static 0.97
--kv-cache-dtype fp8_e4m3
--max-total-tokens 80000
--max-running-requests 2
--attention-backend nsa
--fp8-gemm-backend deep_gemm
--disable-shared-experts-fusion
--tool-call-parser glm47
--reasoning-parser glm45
--served-model-name GLM5.2
--log-level debug
--nsa-prefill-backend flashinfer_sparse_mla
--nsa-decode-backend flashinfer_sparse_mla
```
### Others
_No response_
Contributor guide
Research direction
Start by reviewing the referenced automated install.sh changes and the supplied launch command, then compare the linked sglang and Ktransformers branches with the current repository. The issue does not define a specific code change or acceptance test; clarify whether the goal is installer support, sm120 FP8 support, performance reporting, or future NVFP4 work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100