abetlen / abetlen/llama-cpp-python

implement prompt template for chat completion

Ouverte
#717 37 commentaires 8 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Python
Étoiles
10.6k
Forks
1.4k
Métriques de merge des PR
Métriques de PR en attente

Description

**Is your feature request related to a problem? Please describe.**
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:

```
### User:
### Assistant:
```

system message is currently ignored.

https://github.com/abetlen/llama-cpp-python/blob/255d653ae3bd08c690dbf01f9533daf75f71217c/llama_cpp/llama.py#L1578

This mostly works for most models. But it's not correct.

**Describe the solution you'd like**

1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's [instruction templates](https://github.com/oobabooga/text-generation-webui/blob/main/instruction-templates/Alpaca.yaml) or fastchat's [conversation](https://github.com/lm-sys/FastChat/blob/3149253988ee16b0945aa0a381a42a07b8a7829e/fastchat/conversation.py#L384)
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).

**Describe alternatives you've considered**
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.

**Additional context**

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.