kvcache-ai / kvcache-ai/ktransformers
RTX50系显卡上如何部署?
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
CPU:Intel Xeon w7-3565X (64) @ 4.80 GHz
GPU:NVIDIA GeForce RTX 5090 32G
内存:DDR5-5600 ECC Registered 64G × 16 = 1024G
系统:Ubuntu 22.04.4 LTS x86_64
KTranformers 版本:v0.3.2
CUDA 版本:12.9
运行 DeepSeek-V2-Lite-Chat 模型:
```bash
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./DeepSeek-V2-Lite-Chat-GGUF
```
Hugging Face transformers 的不同版本似乎都无法支持 Kt 运行:
* transformers = 4.43.2
```bash
ImportError: cannot import name 'FlashAttentionKwargs' from 'transformers.modeling_flash_attention_utils' (/home/xiejianan/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/modeling_flash_attention_utils.py)
```
* transformers = 4.56.1
```bash
ValueError: You should provide exactly one of layers or layer_class_to_replicate to initialize a Cache.
```
求问在 RTX50 系显卡上(CUDA版本大于12.8)如何部署 Ktransformers?
Contributor guide
Research direction
Start with the `python -m ktransformers.local_chat` entry point and reproduce the reported DeepSeek-V2-Lite-Chat failures using KTransformers v0.3.2, CUDA 12.9, and the two listed transformers versions. Compare the resulting requirements and errors to determine whether RTX 5090 deployment is supported; done means a verified deployment path or clearly documented incompatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100