kvcache-ai / kvcache-ai/ktransformers

[Bug] ktransformer0.24运行deepseek_q4_km爆内存

Open
#1,510 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

电脑配置是4090*3,cpu是ddr5,512G,系统是ubuntun22.04,
已经成功安装了ktransformer0.24版本,
在运行前已设置了
export CUDA_VISIBLE_DEVICES=0,export USE_NUMA=1,export TORCH_CUDA_ARCH_LIST="8.9",export MAX_JOBS=70
然后运行ktransformer安装命令
python -m ktransformers.local_chat \
--model_path /opt/DeepSeek-V3-0324 \
--gguf_path /opt/DeepSeek-V3-0324-GGUF/Q4_K_M \
--optimize_config_path /opt/ktransformer/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--cpu_infer 70
当加载到40层左右时会爆内存,

### Reproduction

export CUDA_VISIBLE_DEVICES=0,export USE_NUMA=1,export TORCH_CUDA_ARCH_LIST="8.9",export MAX_JOBS=70
python -m ktransformers.local_chat \
--model_path /opt/DeepSeek-V3-0324 \
--gguf_path /opt/DeepSeek-V3-0324-GGUF/Q4_K_M \
--optimize_config_path /opt/ktransformer/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--cpu_infer 70

### Environment

电脑配置是4090*3,cpu是ddr5,512G,系统是ubuntun22.04,

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the command through the ktransformers.local_chat entry point with the stated model paths, DeepSeek-V3-Chat-serve.yaml, and --cpu_infer 70. Trace loading around the reported fortieth layer and compare memory usage on the listed hardware; done means the model loads without exhausting system memory or the issue has a documented, reproducible diagnosis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, ubuntu
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.