kvcache-ai / kvcache-ai/ktransformers
0.3.1 卡在loading model.layers
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
- ubuntu 2204
- 48核CPU
- nvidia 4090 *4
- 128G 内存
- ktransformers 0.3.1+cu126torch27fancy
手动安装了`prometheus-cpp`,`flashinfer`,
```
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
2.7.0+cu126 12.6 True
```
执行下面的命令时没有报错,但是会卡在loading model.layers.12上
```
numactl -N 1 -m 1 python ktransformers/server/main.py --model_path "deepseek-ai/DeepSeek-V3-0324" --gguf_path /home/ubuntu/hub/deepseek-v3-0324 --cpu_infer 49 --optimize_config_path /home/ubuntu/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-4.yaml --port 10002 --chunk_size 4096 --max_new_tokens 1024 --max_batch_size 4 --port 10002 --cache_lens 10240 --backend_type balance_serve
```
以下是命令运行后前面部分的输出
>found flashinfer
found flash_attn
set start method
Connected to server at tcp://localhost:58541
found flashinfer
found flash_attn
试着去掉`numactl -N 1 -m 1 `问题依旧.
当运行到`loading model.layers.12`时观察cpu,内存、GPU的资源占用都非常低了,等待30分钟后仍然卡在loading model那里。

Contributor guide
Assessment
This issue has not been assessed yet.