abetlen / abetlen/llama-cpp-python

implement prompt template for chat completion

未關閉
#717 37 則留言 8 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

**Is your feature request related to a problem? Please describe.**
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:

```
### User:
### Assistant:
```

system message is currently ignored.

https://github.com/abetlen/llama-cpp-python/blob/255d653ae3bd08c690dbf01f9533daf75f71217c/llama_cpp/llama.py#L1578

This mostly works for most models. But it's not correct.

**Describe the solution you'd like**

1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's [instruction templates](https://github.com/oobabooga/text-generation-webui/blob/main/instruction-templates/Alpaca.yaml) or fastchat's [conversation](https://github.com/lm-sys/FastChat/blob/3149253988ee16b0945aa0a381a42a07b8a7829e/fastchat/conversation.py#L384)
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).

**Describe alternatives you've considered**
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.

**Additional context**

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。