abetlen / abetlen/llama-cpp-python

OpenAI compatible server returns empty chat completion

Đang mở
#1,195 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [ x I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [ x I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

Running llama_cpp.server with a Mixtral model can respond to chat completion.

# Current Behavior

Running llama_cpp.server with a Mixtral model does not return any content, it stops immediately.

# Failure Information (for bugs)

I dig some digging and this seems to be caused by the result.stop in function chat_formatter_to_chat_completion_handler.

Here is the value of result:
```
ChatFormatterResponse(prompt='<|im_start|>user\nTranslate this sentence from English to French. I love programming.<|im_end|>\n<|im_start|>user\nTranslate this sentence from English to French. I love programming.<|im_end|>\n<|im_start|>assistant\n', stop=[''])
```

As you can see stop contains an empty string, this seems to be the cause of the issue. When I force stop to None after calling the chat_formatter it solves the issue and I get a proper response.

Also when running with verbose we can see that bos_token and eos_token are empty:
```
Model metadata: {'general.quantization_version': '2', 'tokenizer.chat_template': "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}", 'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.add_bos_token': 'true', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count_kv': '8', 'llama.context_length': '32768', 'llama.attention.head_count': '32', 'llama.rope.freq_base': '1000000.000000', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.model': 'llama', 'llama.expert_used_count': '2', 'general.file_type': '15', 'llama.feed_forward_length': '14336', 'llama.embedding_length': '4096', 'llama.expert_count': '8', 'llama.block_count': '32', 'general.name': 'ehartford_dolphin-2.5-mixtral-8x7b', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'general.architecture': 'llama'}
Using chat template: {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
Using chat eos_token:
Using chat bos_token:
```

# Steps to Reproduce

1) Run llama_cpp.server with any Mixtral model.
2) Make a chat completion request to the server.
3) Server response is empty.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.