kvcache-ai / kvcache-ai/ktransformers
GPTQ_INT4 method crashes with RuntimeError on MoE weight loading (w2 tensor dimension mismatch)
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
Body: $0.00 spent
## Environment
- ktransformers: latest main branch (built from source, Apr 2026) LSP
- GPU: 4x RTX 3090 (SM 86, Ampere) LSPs will activate as files are read
- CPU: AMD EPYC 7443P (AVX2 only)
- Model: Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 ▼ Todo
- kt-method: GPTQ_INT4 [✓] Update MD file with Qwen3.5-122B
- tensor-parallel-size: 4 results and quantization
learnings
## Context [ ] Start Qwen3.5-122B NVFP4 test
We are testing `--kt-method GPTQ_INT4` as a workaround for the [✓] Research self-quantizing MiniMax
known `fp8e4nv not supported` crash on Ampere GPUs (Issue #1930). M2.7 to GPTQ-INT4
The GPTQ_INT4 path successfully avoids the Triton FP8 crash and
correctly selects gptq_marlin kernels. However it fails during
MoE expert weight loading.
## Error
File "sglang/srt/layers/moe/fused_moe_triton/layer.py", line 518, in _load_w2
loaded_weight = loaded_weight.narrow(
RuntimeError: start (8) + length (8) exceeds dimension size (8).
## Launch command
```bash
python -m sglang.launch_server \
--model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
--kt-weight-path Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
--kt-method GPTQ_INT4 \
--tensor-parallel-size 4 \
--kt-num-gpu-experts 30 \
--kt-cpuinfer 24 \
--attention-backend flashinfer \
--disable-shared-experts-fusion
Note
This is important because GPTQ_INT4 with Marlin kernels is
currently the only viable kt-method for Ampere (SM 86) GPUs,
since FP8 requires SM 89+ and AMXINT4/8 requires Intel AMX.
Fixing this would unlock ktransformers for the large RTX 3090
user base.
### Reproduction
```text
Put your message here.
```
### Others
_No response_
Contributor guide
Research direction
Start with sglang/srt/layers/moe/fused_moe_triton/layer.py at _load_w2, where the RuntimeError occurs during MoE expert weight loading. Reproduce the failure with the provided Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 launch command and inspect the tensor dimensions around the narrow operation. Done means the model loads successfully with GPTQ_INT4 and the reported dimension-mismatch crash no longer occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100