kvcache-ai / kvcache-ai/ktransformers

0.3.1 卡在loading model.layers

Open
#1,367 3 comments 0 reactions 1 assignee Claimed by @ovowei View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

- ubuntu 2204
- 48核CPU
- nvidia 4090 *4
- 128G 内存
- ktransformers 0.3.1+cu126torch27fancy

手动安装了`prometheus-cpp`,`flashinfer`,

```
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
2.7.0+cu126 12.6 True
```

执行下面的命令时没有报错,但是会卡在loading model.layers.12上

```
numactl -N 1 -m 1 python ktransformers/server/main.py --model_path "deepseek-ai/DeepSeek-V3-0324" --gguf_path /home/ubuntu/hub/deepseek-v3-0324 --cpu_infer 49 --optimize_config_path /home/ubuntu/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4.yaml --port 10002 --chunk_size 4096 --max_new_tokens 1024 --max_batch_size 4 --port 10002 --cache_lens 10240 --backend_type balance_serve
```
以下是命令运行后前面部分的输出
>found flashinfer
found flash_attn
set start method
Connected to server at tcp://localhost:58541
found flashinfer
found flash_attn

试着去掉`numactl -N 1 -m 1 `问题依旧.
当运行到`loading model.layers.12`时观察cpu,内存、GPU的资源占用都非常低了,等待30分钟后仍然卡在loading model那里。

![Image](https://github.com/user-attachments/assets/3c042554-7676-4b7e-89e0-70cf518fe10b)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.