kvcache-ai / kvcache-ai/ktransformers

[Feature Request] qwen4_exp support (Qwen3.8-Flash-Next) — CPU/GPU hybrid expert offload via kt-kernel/sglang-kt

Open
#2,179 0 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### Description

Model: [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) (HF model_type: qwen4_exp), released 2026-08-26. Qwen4 architecture preview: 125B main model, 6B active/token, 48 layers, MoE with 512 routed experts + 1 shared, top-10 routing; plus 51B N-gram embedding table (PLE) and 4B MTP module.

Request: kt-kernel expert-offload support for qwen4_exp, analogous to the existing Qwen3-Next / Qwen3-Coder-Next sglang-kt integrations.

Why this model fits KTransformers particularly well:

- Ultra-sparse routing (10/512) means a hotness-aware GPU-resident expert set (--kt-num-gpu-experts + --kt-enable-dynamic-expert-update) should capture most expert traffic — the skew is stronger than in Qwen3MoE.
- The PLE table already has a host-offload story in vanilla SGLang; kt-kernel expert offload is the missing piece that makes single-GPU / small-VRAM serving of this model viable at FP8 expert precision instead of INT4 requantization.
- Qwen's own model card recommends KTransformers for production serving of this model, but no qwen4_exp support exists yet.

Architecture deltas vs. Qwen3-Next that affect integration (from the model card / SGLang PRs #36497, #36585):

- GDN layers carried over (sigmoid output gate instead of silu per llama.cpp PR #27742 notes); 36 GDN + 12 QSA sparse-attention layers
- QSA: lightweight indexer selecting context at micro-block granularity (512 blocks / 2048-token budget)
- Gated Residual: 4-branch residual stream, low-rank (r=320) read/write gates
- PLE n-gram embedding input path at layer 2
- MTP draft layers use QSA as well

The SGLang model graph is landing in sgl-project/sglang#36497 / #36585, so the injection target should exist shortly.

Hardware I can test on: dual Xeon Gold 6526Y (Emerald Rapids, AMX), 512 GB DDR5, 2× RTX 6000 Ada (48 GB, SM89) — and separately a Ryzen 9950X + RTX 5080 (SM120) consumer box. Happy to run benchmarks/validation on both AMX and non-AMX paths, single- and dual-GPU, for a multi-user serving workload (~8 concurrent).

### Pull Request

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing Qwen3-Next and Qwen3-Coder-Next sglang-kt integrations with the Qwen4 architecture described in the model card and SGLang PRs #36497 and #36585. Confirm where the qwen4_exp model graph lands, then define the integration scope for expert offload, GDN, QSA, gated residual, PLE, and MTP paths. Done should include working single- and dual-GPU serving plus validation on the listed AMX and non-AMX hardware.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.