kvcache-ai / kvcache-ai/ktransformers

[Bug] 部署unsloth/DeepSeek-R1-0528-GGUF/UD-Q8_K_XL 报错

Open
#1,382 10 comments 0 reactions 1 assignee Claimed by @Azure-Tang View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### 检查清单

- [x] 1. 我已经搜索过相关问题,但未能获得预期的帮助
- [x] 2. 该问题在最新版本中尚未修复
- [x] 3. 请注意,如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例,我们将难以复现和定位问题,降低获得反馈的可能性
- [x] 4. 如果您提出的不是bug而是问题,请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
- [x] 5. 为方便社区交流,我将使用中文/英文或附上中文/英文翻译(如使用其他语言)。未附带翻译的非中文/英语内容可能会被关闭

### 问题描述

报错:
```
/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:112: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
```

### 复现步骤

命令:
```
python3 ktransformers/server/main.py \
--gguf_path /mnt/nvme0/models/unsloth/DeepSeek-R1-0528-GGUF/UD-Q8_K_XL/ \
--model_path deepseek-ai/DeepSeek-R1 \
--model_name unsloth/DeepSeek-R1-0528-GGUF \
--cpu_infer 128 \
--max_new_tokens 8192 \
--cache_lens 32768 \
--temperature 0.6 \
--top_p 0.95 \
--optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-8.yaml \
--host 127.0.0.1 \
--port 1888
```
模型:
unsloth/DeepSeek-R1-0528-GGUF:UD-Q8_K_XL
https://modelscope.cn/models/unsloth/DeepSeek-R1-0528-GGUF/files

### 环境信息

操作系统:Ubuntu 22.04
GPU: A800-SXM4-80GB * 8
CPU: Intel(R) Xeon(R) Platinum 8338C CPU @ 2.60GHz

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.