abetlen / abetlen/llama-cpp-python

implement prompt template for chat completion

Offen
#717 37 Kommentare 8 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
10.6k
Forks
1.4k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

**Is your feature request related to a problem? Please describe.**
When generating chat completion, it is hard-coded to generate a non-standard prompt template that looks something like:

```
### User:
### Assistant:
```

system message is currently ignored.

https://github.com/abetlen/llama-cpp-python/blob/255d653ae3bd08c690dbf01f9533daf75f71217c/llama_cpp/llama.py#L1578

This mostly works for most models. But it's not correct.

**Describe the solution you'd like**

1. add a set of built-in prompt templates user can specify at inference time ["vicuna","alpaca","chatml","llama2-chat","oasst"] at minimum
2. recommend copying design from ooba's [instruction templates](https://github.com/oobabooga/text-generation-webui/blob/main/instruction-templates/Alpaca.yaml) or fastchat's [conversation](https://github.com/lm-sys/FastChat/blob/3149253988ee16b0945aa0a381a42a07b8a7829e/fastchat/conversation.py#L384)
3. add ability to pass a template string for other nonstandard formats (such as the one currently implemented in llama-cpp-python).

**Describe alternatives you've considered**
modifying llama-cpp-python to hard code it to llama2-chat format, not a great solution.

**Additional context**

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.