abetlen / abetlen/llama-cpp-python

implement prompt template for chat completion

Aperta
#717 37 commenti 8 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Python
Stelle
10.6k
Fork
1.4k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

**Is your feature request related to a problem? Please describe.**
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:

```
### User:
### Assistant:
```

system message is currently ignored.

https://github.com/abetlen/llama-cpp-python/blob/255d653ae3bd08c690dbf01f9533daf75f71217c/llama_cpp/llama.py#L1578

This mostly works for most models. But it's not correct.

**Describe the solution you'd like**

1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's [instruction templates](https://github.com/oobabooga/text-generation-webui/blob/main/instruction-templates/Alpaca.yaml) or fastchat's [conversation](https://github.com/lm-sys/FastChat/blob/3149253988ee16b0945aa0a381a42a07b8a7829e/fastchat/conversation.py#L384)
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).

**Describe alternatives you've considered**
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.

**Additional context**

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.