kvcache-ai / kvcache-ai/ktransformers

RTX50系显卡上如何部署?

Open
#1,509 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

CPU:Intel Xeon w7-3565X (64) @ 4.80 GHz
GPU:NVIDIA GeForce RTX 5090 32G
内存:DDR5-5600 ECC Registered 64G × 16 = 1024G
系统:Ubuntu 22.04.4 LTS x86_64
KTranformers 版本:v0.3.2
CUDA 版本:12.9
运行 DeepSeek-V2-Lite-Chat 模型:
```bash
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Lite-Chat --gguf_path ./DeepSeek-V2-Lite-Chat-GGUF
```

Hugging Face transformers 的不同版本似乎都无法支持 Kt 运行:
* transformers = 4.43.2
```bash
ImportError: cannot import name 'FlashAttentionKwargs' from 'transformers.modeling_flash_attention_utils' (/home/xiejianan/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/modeling_flash_attention_utils.py)
```
* transformers = 4.56.1
```bash
ValueError: You should provide exactly one of layers or layer_class_to_replicate to initialize a Cache.
```

求问在 RTX50 系显卡上(CUDA版本大于12.8)如何部署 Ktransformers?

Contributor guide

Open the contributing guide

Research direction

Start with the `python -m ktransformers.local_chat` entry point and reproduce the reported DeepSeek-V2-Lite-Chat failures using KTransformers v0.3.2, CUDA 12.9, and the two listed transformers versions. Compare the resulting requirements and errors to determine whether RTX 5090 deployment is supported; done means a verified deployment path or clearly documented incompatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.