kvcache-ai / kvcache-ai/ktransformers

[Bug] 为什么 Qwen2-57B-GGUF 的chat结果是空白?

Open
#1,532 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19.5k
Forks
1.6k
Avg merge
19h 32m
Merged PRs (30d)
27

Description

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

### Describe the bug

大家好,

我通过这个命令启动了大模型:
python3 ktransformers/ktransformers/local_chat.py --model_path ./Qwen2-57B-A14B-Instruct --gguf_path Qwen2-57B-GGUF

然后进入到这个界面:
Chat:

然后我输入了问题:
1+1=?

然后显示如下:
```
Chat: 1+1=? (下面不是空行,是很多空白)

prompt eval count: 23 token(s)
prompt eval duration: 1.0584766864776611s
prompt eval rate: 21.729340186544967 tokens/s
eval count: 1000 token(s)
eval duration: 102.8101806640625s
eval rate: 9.726663191727587 tokens/s
Chat:
```

这是为什么?是模型输出的问题?还是ktransformers编解码的问题?请问有谁用过这个模型吗?

### Reproduction

```
python3 ktransformers/ktransformers/local_chat.py --model_path ./Qwen2-57B-A14B-Instruct --gguf_path Qwen2-57B-GGUF
```

### Environment

```
Chat: 1+1=? (下面不是空行,是很多空白)

prompt eval count: 23 token(s)
prompt eval duration: 1.0584766864776611s
prompt eval rate: 21.729340186544967 tokens/s
eval count: 1000 token(s)
eval duration: 102.8101806640625s
eval rate: 9.726663191727587 tokens/s
Chat:
```

Contributor guide

Open the contributing guide

Research direction

Start by running the reproduction command through ktransformers/local_chat.py with the Qwen2-57B-GGUF model and compare the reported blank output with the issue details. Investigate whether the empty response comes from the model output or ktransformers encoding, and consider the issue resolved when the chat produces visible output for the same prompt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.