kvcache-ai / kvcache-ai/ktransformers

[Bug] 使用AMX,运行Qwen3Moe-30B卡在warm up,”capturing cuda graph 1 1“处

Open
#1,486 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

使用AMX,运行Qwen3Moe-30B卡在warm up,”capturing cuda graph 1 1“处,
loading model.layers.47.self_attn.k_norm.weight to cuda
loading model.layers.47.input_layernorm.weight to cuda
loading model.layers.47.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 3027579
Got inference context, sending it to subscribers.
Rebuilding kvcache
48
kv_cache loaded successfully.
capturing cuda graph 1 1
2025-08-28 09:33:05,319 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-08-28 09:33:05,331 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-08-28 09:33:05,762 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-08-28 09:33:05,770 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-08-28 09:33:05,905 - INFO - flashinfer.jit: Loading JIT ops: page
2025-08-28 09:33:05,914 - INFO - flashinfer.jit: Finished loading JIT ops: page

一直卡住。。。。。。。。
如果使用的是 Qwen3Moe-serve.yaml能够正常运行

### Reproduction

命令:
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM --model_path Qwen3-30B-A3B-bf16-JSON --gguf_path Qwen3-30B-A3B-bf16-GGUF --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve-amx.yaml --backend_type balance_serve
模型 Qwen3-30B

### Environment

系统: Ubuntu 20.04
CPU: Intel Xeon Gold 6434
GPU: 4090

Contributor guide

Open the contributing guide

Research direction

Start with ktransformers/server/main.py and reproduce the command using Qwen3Moe-serve-amx.yaml on the listed Ubuntu, CPU, and GPU setup. Compare that configuration with Qwen3Moe-serve.yaml and trace the warm-up output after “capturing cuda graph 1 1”; done means the AMX configuration completes warm-up or the hang has a documented, reproducible cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.