kvcache-ai / kvcache-ai/ktransformers
[Feature Request] qwen4_exp support (Qwen3.8-Flash-Next) — CPU/GPU hybrid expert offload via kt-kernel/sglang-kt
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### Description
Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) (HF model_type: qwen4_exp), released 2026-08-26. Qwen4 architecture preview: 125B main model, 6B active/token, 48 layers, MoE with 512 routed experts + 1 shared, top-10 routing; plus 51B N-gram embedding table (PLE) and 4B MTP module.
Request: kt-kernel expert-offload support for qwen4_exp, analogous to the existing Qwen3-Next / Qwen3-Coder-Next sglang-kt integrations.
Why this model fits KTransformers particularly well:
- Ultra-sparse routing (10/512) means a hotness-aware GPU-resident expert set (--kt-num-gpu-experts + --kt-enable-dynamic-expert-update) should capture most expert traffic — the skew is stronger than in Qwen3MoE.
- The PLE table already has a host-offload story in vanilla SGLang; kt-kernel expert offload is the missing piece that makes single-GPU / small-VRAM serving of this model viable at FP8 expert precision instead of INT4 requantization.
- Qwen's own model card recommends KTransformers for production serving of this model, but no qwen4_exp support exists yet.
Architecture deltas vs. Qwen3-Next that affect integration (from the model card / SGLang PRs #36497, #36585):
- GDN layers carried over (sigmoid output gate instead of silu per llama.cpp PR #27742 notes); 36 GDN + 12 QSA sparse-attention layers
- QSA: lightweight indexer selecting context at micro-block granularity (512 blocks / 2048-token budget)
- Gated Residual: 4-branch residual stream, low-rank (r=320) read/write gates
- PLE n-gram embedding input path at layer 2
- MTP draft layers use QSA as well
The SGLang model graph is landing in sgl-project/sglang#36497 / #36585, so the injection target should exist shortly.
Hardware I can test on: dual Xeon Gold 6526Y (Emerald Rapids, AMX), 512 GB DDR5, 2× RTX 6000 Ada (48 GB, SM89) — and separately a Ryzen 9950X + RTX 5080 (SM120) consumer box. Happy to run benchmarks/validation on both AMX and non-AMX paths, single- and dual-GPU, for a multi-user serving workload (~8 concurrent).
### Pull Request
_No response_
Contributor guide
Research direction
Start by comparing the existing Qwen3-Next and Qwen3-Coder-Next sglang-kt integrations with the Qwen4 architecture described in the model card and SGLang PRs #36497 and #36585. Confirm where the qwen4_exp model graph lands, then define the integration scope for expert offload, GDN, QSA, gated residual, PLE, and MTP paths. Done should include working single- and dual-GPU serving plus validation on the listed AMX and non-AMX hardware.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100