abetlen / abetlen/llama-cpp-python
implement prompt template for chat completion
- Lingua principale
- Python
- Stelle
- 10.6k
- Fork
- 1.4k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
**Is your feature request related to a problem? Please describe.**
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:
```
### User:
### Assistant:
```
system message is currently ignored.
https://github.com/abetlen/llama-cpp-python/blob/255d653ae3bd08c690dbf01f9533daf75f71217c/llama_cpp/llama.py#L1578
This mostly works for most models. But it's not correct.
**Describe the solution you'd like**
1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's [instruction templates](https://github.com/oobabooga/text-generation-webui/blob/main/instruction-templates/Alpaca.yaml) or fastchat's [conversation](https://github.com/lm-sys/FastChat/blob/3149253988ee16b0945aa0a381a42a07b8a7829e/fastchat/conversation.py#L384)
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).
**Describe alternatives you've considered**
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.
**Additional context**
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.