kvcache-ai / kvcache-ai/ktransformers
[Bug] ktransformer0.24运行deepseek_q4_km爆内存
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
电脑配置是4090*3,cpu是ddr5,512G,系统是ubuntun22.04,
已经成功安装了ktransformer0.24版本,
在运行前已设置了
export CUDA_VISIBLE_DEVICES=0,export USE_NUMA=1,export TORCH_CUDA_ARCH_LIST="8.9",export MAX_JOBS=70
然后运行ktransformer安装命令
python -m ktransformers.local_chat \
--model_path /opt/DeepSeek-V3-0324 \
--gguf_path /opt/DeepSeek-V3-0324-GGUF/Q4_K_M \
--optimize_config_path /opt/ktransformer/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--cpu_infer 70
当加载到40层左右时会爆内存,
### Reproduction
export CUDA_VISIBLE_DEVICES=0,export USE_NUMA=1,export TORCH_CUDA_ARCH_LIST="8.9",export MAX_JOBS=70
python -m ktransformers.local_chat \
--model_path /opt/DeepSeek-V3-0324 \
--gguf_path /opt/DeepSeek-V3-0324-GGUF/Q4_K_M \
--optimize_config_path /opt/ktransformer/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml \
--cpu_infer 70
### Environment
电脑配置是4090*3,cpu是ddr5,512G,系统是ubuntun22.04,
Contributor guide
Research direction
Start by reproducing the command through the ktransformers.local_chat entry point with the stated model paths, DeepSeek-V3-Chat-serve.yaml, and --cpu_infer 70. Trace loading around the reported fortieth layer and compare memory usage on the listed hardware; done means the model loads without exhausting system memory or the issue has a documented, reproducible diagnosis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, ubuntu
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100