kvcache-ai / kvcache-ai/ktransformers

GLM5.2 on RTX 6000 Pro ( sm120 ) * Update

Open
#2,083 1 comment 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

Hi,

Just a little update on Kt-kernel FP8 sm120 support. I managed to get it working by having codex backport sm120 support onto the kvcache-ai/sglang branch ( based off of this https://github.com/NVIDIA/nvidia-gcp-samples/pull/25 ). You can find that branch at https://github.com/trilog-inc/sglang/tree/codex/backport-sm120-glm52-kt . There is an accompanying Ktransformers fork https://github.com/trilog-inc/ktransformers/tree/codex/sm120-glm52-install-docs. The changes are mostly on the automated install.sh script to explicitly set the sm120 arch and to point the sglang install to a local directory instead of the kvcache branch.

A few changes to the nsa backend had to be done in order to accommodate the new flashinfer and deep_gemm versions, however the fix is pretty straightforward. The heavy lifting is being done by the updated flashinfer version (post-0.6.13) and the nv_dev branch of DeepGEMM ( nv_dev @ a6b593d ). Make sure to recompile the sgl-kernel once the nv_dev branch of deepGemm is pulled and built.

On a w7-3465x Xeon with 768GB DDR5 @ 5600 ( AMX ) and a single RTX 6000 Pro, I get prompt processing speeds of ~120t/s and generation of about 8t/s. I ran an entire OpenCode session and everything looks good.

If anyone is curious to try, please report performance. The next step would be to work out NVFP4 for increased performance but nominal intelligence drop.

### Reproduction

Launch command:

```text
SGLANG_ENABLE_JIT_DEEPGEMM=1 FLASHINFER_CUDA_ARCH_LIST=12.0f CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server
--model-path /mnt/home_extend/models/GLM52
--kt-weight-path /mnt/home_extend/models/GLM52
--kt-cpuinfer 50
--kt-threadpool-count 2
--kt-num-gpu-experts 20
--kt-method FP8
--kt-gpu-prefill-token-threshold 1024
--kt-enable-dynamic-expert-update
--kt-expert-placement-strategy uniform
--tp-size 1
--trust-remote-code
--host 0.0.0.0
--port 60000
--mem-fraction-static 0.97
--kv-cache-dtype fp8_e4m3
--max-total-tokens 80000
--max-running-requests 2
--attention-backend nsa
--fp8-gemm-backend deep_gemm
--disable-shared-experts-fusion
--tool-call-parser glm47
--reasoning-parser glm45
--served-model-name GLM5.2
--log-level debug
--nsa-prefill-backend flashinfer_sparse_mla
--nsa-decode-backend flashinfer_sparse_mla
```

### Others

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the referenced automated install.sh changes and the supplied launch command, then compare the linked sglang and Ktransformers branches with the current repository. The issue does not define a specific code change or acceptance test; clarify whether the goal is installer support, sm120 FP8 support, performance reporting, or future NVFP4 work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.