kvcache-ai / kvcache-ai/ktransformers

FP8 crashes on sm_86 (RTX 3090) even with --kt-num-gpu-experts 0

Open
#1,930 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

## `--kt-method FP8` crashes on sm_86 (RTX 3090) even with `--kt-num-gpu-experts 0`

### Environment
- GPU: RTX 3090 (sm_86)
- CPU: AMD EPYC 7663 (AVX2)
- KTransformers: v0.5.3
- sgl-kernel: 0.3.21
- Model: `Qwen/Qwen3.5-122B-A10B-FP8`

### Steps to reproduce

```
python -m sglang.launch_server \
--model /path/to/Qwen3.5-122B-A10B-FP8 \
--kt-weight-path /path/to/Qwen3.5-122B-A10B-FP8 \
--kt-method FP8 \
--kt-cpuinfer 56 \
--kt-threadpool-count 1 \
--kt-num-gpu-experts 0 \
--attention-backend triton \
--tensor-parallel-size 2 \
--trust-remote-code \
--disable-shared-experts-fusion \
--disable-custom-all-reduce
```

### What happens

AVX2 FP8 MoE layers initialize successfully on CPU:
```
TP MOE layer 47, pool: 0x6512e0e0, expert num: 256, num_experts_per_tok: 8
Created AVX2_FP8_MOE_TP 0 at numa 0
```

Model loads, KV cache allocates, then during CUDA graph capture:
```
triton.compiler.errors.CompilationError: at 1:0:
def fused_moe_kernel(
^
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
```

Setting `--moe-runner-backend cutlass` hits:
```
AssertionError: cutlass_fp8 MoE requires CUDA 12.0+ with SM90 or CUDA 12.4+ with SM89
```

### Notes

- `--kt-num-gpu-experts` is set to 0
- The AVX2 FP8 CPU kernels load and initialize without error
- The crash occurs in the GPU-side MoE kernel compilation path
- `--kt-method BF16` does not crash
- RTX 3090 (sm_86) supports `fp8e4b15` and `fp8e5` per the Triton error message, but not `fp8e4nv`
- The [AVX2 tutorial](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/AVX2-Tutorial.md) lists RTX 3090 as supported hardware and includes an FP8 example

### Related
- #934

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the command with --kt-method FP8 and --kt-num-gpu-experts 0, then trace the GPU-side MoE compilation path reached during CUDA graph capture. Compare the failing fp8e4nv selection with the FP8 types supported on sm_86 and the cutlass backend restriction. Done means the RTX 3090 path no longer crashes, or reports a clear unsupported-configuration error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.