kvcache-ai/ktransformers

View on GitHub

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Stars: 19.5k
Forks: 1.6k
Open beginner issues: 1
Indexed issues: 465
Avg merge: 19h 32m
Merged PRs (30d): 27

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 19, 2026
Latest indexed: Sep 18, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue help wanted

Beginner friendly Everything

Open All

465 open issues indexed

A100 not support? Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 25/100

kvcache-ai/ktransformers#2208 · Opened Sep 15, 2026
cache_salt/extra_key are computed & carried but never delivered to Req — radix-cache tenant isolation is a no-op (regression vs upstream sglang) Open

Difficulty 3/5 1-2 days Newbie friendliness 76/100

kvcache-ai/ktransformers#2184 · 2 comments · Opened Sep 2, 2026
[Feature Request] qwen4_exp support (Qwen3.8-Flash-Next) — CPU/GPU hybrid expert offload via kt-kernel/sglang-kt Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

kvcache-ai/ktransformers#2179 · 5 reactions · Opened Aug 28, 2026
[MXFP4 layerwise prefill] SM 8.6 (Ampere) excluded by capability whitelist — DSV4-Flash on dual A10 falls back to serialized full-GPU path Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#2172 · 2 comments · Opened Aug 26, 2026
GeneralMoEWrapper.load_weights() cpu_save path reads nonexistent dict keys, guaranteed KeyError Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

kvcache-ai/ktransformers#2151 · 1 comment · Opened Aug 9, 2026
kt-sft: chunked_prefill_size sized from cutoff_len, never scaled by batch — any per_device_train_batch_size>1 hard-errors Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

kvcache-ai/ktransformers#2150 · 1 comment · Opened Aug 8, 2026
When will kvcache‑ai/sglang be synced with sgl‑project/sglang Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

kvcache-ai/ktransformers#2139 · 3 reactions · Opened Aug 5, 2026
DeepSeek-V4-Flash-0731 MTP failed to start on avx2 + sm89 Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#2127 · 3 comments · 2 reactions · Opened Aug 3, 2026
Support DSpark speculative decoding for DeepSeek-V4-Flash-0731 in sglang-kt Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

kvcache-ai/ktransformers#2118 · 1 comment · 3 reactions · Opened Aug 2, 2026
ROCm Support for Dual AMD Radeon Pro V620 (GLM-5.2) Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2115 · 1 reaction · Opened Jul 31, 2026
MiniMax M2.7: AttributeError: 'LlamafileMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer' Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#2113 · 3 comments · Opened Jul 31, 2026
kimi k3啥时候支持？ Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#2109 · 1 comment · Opened Jul 28, 2026
GLM5.2 int8: 'AMXMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer' Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

kvcache-ai/ktransformers#2108 · 1 comment · 1 reaction · Opened Jul 27, 2026
RuntimeError: Only Tensors of floating point and complex dtype can require gradients Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

kvcache-ai/ktransformers#2107 · 1 reaction · Opened Jul 26, 2026
Error with Deepskeep 4 flash with trtllm_fp4_block_scale_moe Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

kvcache-ai/ktransformers#2106 · 1 comment · Opened Jul 23, 2026
[Bug] 无法正确识别模型文件 Open

Difficulty 3/5 1-2 days Newbie friendliness 52/100

kvcache-ai/ktransformers#2100 · Opened Jul 21, 2026
kt-kernel/install.sh failed with A100 Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

kvcache-ai/ktransformers#2098 · Opened Jul 20, 2026
glm5.2 slow on 4*H100 Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

kvcache-ai/ktransformers#2090 · 1 comment · Opened Jul 17, 2026
kt-kernel loads every expert into CPU RAM even when they're GPU-resident (follow-up to #2023) Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

kvcache-ai/ktransformers#2084 · 3 comments · Opened Jul 12, 2026
GLM5.2 on RTX 6000 Pro ( sm120 ) * Update Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#2083 · 1 comment · 4 reactions · Opened Jul 10, 2026
[Bug] Kimi-K2.6 RAWINT4: corrupted output with `--tensor-parallel-size 4`, correct output with TP1 — same weights, same build (4× H200 NVL, AMD EPYC 9555) Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#2076 · 3 comments · Opened Jul 7, 2026
openPangu 2.0 Flash support Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 30/100

kvcache-ai/ktransformers#2066 · Opened Jun 30, 2026
KT+SGLANG 部署 GLM5.2-FP8出现 `Python error: Segmentation fault` Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2061 · 3 comments · Opened Jun 25, 2026
[Feature] Support repacking Intel Autoround format(s) into AMXINT4/AMXINT8 Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

kvcache-ai/ktransformers#2060 · 1 comment · Opened Jun 24, 2026
unauthenticated `pickle.loads` on TCP ZMQ socket allows pre-auth RCE in `balance_serve` scheduler Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#2042 · Opened Jun 16, 2026
部署Qwen3.6-35B-A3B-FP8出现的问题 Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2040 · 4 comments · Opened Jun 12, 2026
# [Bug/Workaround] FlashInfer & Triton JIT compilation failure on heterogeneous GPUs (e.g., RTX 3090 + 4090) in TP mode Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

kvcache-ai/ktransformers#2039 · Opened Jun 11, 2026
It seems that GLM 4.5/4.6/4.7 is not yet supported? Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

kvcache-ai/ktransformers#2038 · Opened Jun 9, 2026
KT +SGLANG是否支持GGUF MOE模型读取，例如QWEN3.6 35B A3B GGUF 的 UNSLOTH量化 Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

kvcache-ai/ktransformers#2034 · Opened Jun 5, 2026
small model and colab t4 Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2024 · Opened May 26, 2026
H200 部署 GLM 5.1 Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2019 · 2 comments · Opened May 21, 2026
昇腾NPU在最新版本里面的支持 Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#2012 · Opened May 18, 2026
按照适配NPU的文档无法完成kt的安装 Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2011 · Opened May 18, 2026
Crash when enable MTP on KT 0.6.2 DeepSeekV4Flash Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#2009 · 2 comments · Opened May 16, 2026
Speculative/EAGLE - Supported? Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

kvcache-ai/ktransformers#2007 · 2 comments · Opened May 16, 2026
Discussion on the cost-effectiveness of AMX and FP8 Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

kvcache-ai/ktransformers#2004 · Opened May 14, 2026
支持 SGLang 的 expert-parallel-size（EP） Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#2002 · Opened May 13, 2026
[Help] Possible to run deepseek4-flash on ampere (3090) + threadripper 3970x (no AMX/AVX512) ? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#1999 · 15 comments · Opened May 11, 2026
[Bug] Qwen3.5-397B-A17B-FP8 fails to load: expert_data shape mismatch (2048 vs 4096) at _load_w13, even with TP=1 + AVX2 backend Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

kvcache-ai/ktransformers#1996 · Opened May 10, 2026
Minimax2.5 TypeError: non-default argument 'quantization_config' follows default argument Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

kvcache-ai/ktransformers#1992 · 2 comments · Opened May 9, 2026
24G card encounter error of trying to allocate 32G vram Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

kvcache-ai/ktransformers#1990 · 1 comment · Opened May 9, 2026
5090D 32G single card encounter error assert num_stages >= 1 on first request Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#1988 · 6 comments · Opened May 8, 2026
Common Problem / Fix: Running deepseek v4 on L40 *4, AssertionError occurs in swa_radix_cache.py:565 Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

kvcache-ai/ktransformers#1985 · 17 comments · Opened May 7, 2026
Duplicate BF16SafeTensorLoader overrides Qwen3.5 BF16 packed MoE loader Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

kvcache-ai/ktransformers#1983 · 1 comment · Opened May 7, 2026
Kimi K2.6 works only with the latest reasoning/tool parser from sglang main and fp8-gemm-backend triton Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

kvcache-ai/ktransformers#1961 · 1 comment · Opened Apr 28, 2026
v0.6.1 does not compile SM89 (Ada Lovelace) kernels unlike v0.5.3 Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

kvcache-ai/ktransformers#1956 · 11 comments · Opened Apr 26, 2026
GPTQ_INT4 method crashes with RuntimeError on MoE weight loading (w2 tensor dimension mismatch) Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

kvcache-ai/ktransformers#1943 · 2 comments · Opened Apr 23, 2026
kt-numa-nodes 配置单numa运行，cpu，gpu均按照numa切分，ttft耗时极高 Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

kvcache-ai/ktransformers#1938 · 3 comments · Opened Apr 21, 2026
Pipeline Parallel for ktransformers for multi low-vram GPUs Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

kvcache-ai/ktransformers#1933 · 1 comment · Opened Apr 16, 2026
FP8 crashes on sm_86 (RTX 3090) even with --kt-num-gpu-experts 0 Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

kvcache-ai/ktransformers#1930 · 1 comment · Opened Apr 13, 2026

kvcache-ai/ktransformers

Get new issues in your inbox