kvcache-ai / kvcache-ai/ktransformers

GPTQ_INT4 method crashes with RuntimeError on MoE weight loading (w2 tensor dimension mismatch)

Open
#1,943 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

Body: $0.00 spent
## Environment
- ktransformers: latest main branch (built from source, Apr 2026) LSP
- GPU: 4x RTX 3090 (SM 86, Ampere) LSPs will activate as files are read
- CPU: AMD EPYC 7443P (AVX2 only)
- Model: Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 ▼ Todo
- kt-method: GPTQ_INT4 [✓] Update MD file with Qwen3.5-122B
- tensor-parallel-size: 4 results and quantization
learnings
## Context [ ] Start Qwen3.5-122B NVFP4 test
We are testing `--kt-method GPTQ_INT4` as a workaround for the [✓] Research self-quantizing MiniMax
known `fp8e4nv not supported` crash on Ampere GPUs (Issue #1930). M2.7 to GPTQ-INT4

The GPTQ_INT4 path successfully avoids the Triton FP8 crash and
correctly selects gptq_marlin kernels. However it fails during
MoE expert weight loading.

## Error

File "sglang/srt/layers/moe/fused_moe_triton/layer.py", line 518, in _load_w2
loaded_weight = loaded_weight.narrow(
RuntimeError: start (8) + length (8) exceeds dimension size (8).

## Launch command
```bash
python -m sglang.launch_server \
--model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
--kt-weight-path Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
--kt-method GPTQ_INT4 \
--tensor-parallel-size 4 \
--kt-num-gpu-experts 30 \
--kt-cpuinfer 24 \
--attention-backend flashinfer \
--disable-shared-experts-fusion

Note
This is important because GPTQ_INT4 with Marlin kernels is
currently the only viable kt-method for Ampere (SM 86) GPUs,
since FP8 requires SM 89+ and AMXINT4/8 requires Intel AMX.
Fixing this would unlock ktransformers for the large RTX 3090
user base.

### Reproduction

```text
Put your message here.
```

### Others

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with sglang/srt/layers/moe/fused_moe_triton/layer.py at _load_w2, where the RuntimeError occurs during MoE expert weight loading. Reproduce the failure with the provided Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 launch command and inspect the tensor dimensions around the narrow operation. Done means the model loads successfully with GPTQ_INT4 and the reported dimension-mismatch crash no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.