kvcache-ai / kvcache-ai/ktransformers
[Bug] 为什么 Qwen2-57B-GGUF 的chat结果是空白?
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Checklist
- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
- [ ] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
- [ ] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.
### Describe the bug
大家好,
我通过这个命令启动了大模型:
python3 ktransformers/ktransformers/local_chat.py --model_path ./Qwen2-57B-A14B-Instruct --gguf_path Qwen2-57B-GGUF
然后进入到这个界面:
Chat:
然后我输入了问题:
1+1=?
然后显示如下:
```
Chat: 1+1=? (下面不是空行,是很多空白)
prompt eval count: 23 token(s)
prompt eval duration: 1.0584766864776611s
prompt eval rate: 21.729340186544967 tokens/s
eval count: 1000 token(s)
eval duration: 102.8101806640625s
eval rate: 9.726663191727587 tokens/s
Chat:
```
这是为什么?是模型输出的问题?还是ktransformers编解码的问题?请问有谁用过这个模型吗?
### Reproduction
```
python3 ktransformers/ktransformers/local_chat.py --model_path ./Qwen2-57B-A14B-Instruct --gguf_path Qwen2-57B-GGUF
```
### Environment
```
Chat: 1+1=? (下面不是空行,是很多空白)
prompt eval count: 23 token(s)
prompt eval duration: 1.0584766864776611s
prompt eval rate: 21.729340186544967 tokens/s
eval count: 1000 token(s)
eval duration: 102.8101806640625s
eval rate: 9.726663191727587 tokens/s
Chat:
```
Contributor guide
Research direction
Start by running the reproduction command through ktransformers/local_chat.py with the Qwen2-57B-GGUF model and compare the reported blank output with the issue details. Investigate whether the empty response comes from the model output or ktransformers encoding, and consider the issue resolved when the chat produces visible output for the same prompt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100