abetlen / abetlen/llama-cpp-python

Support for a limited vocabulary for generation

Open
#998 7 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
I would like to constrain the model output to only use a custom vocabulary comprising a list of allowable words (or alternatively, to blacklist all other words in the vocabulary).

**Describe the solution you'd like**
HuggingFace's transformer library features a `bad_words_id` keyword in the `model.generate` function that accepts a list of words to exclude from its output (some discussion of this feature [here](https://github.com/huggingface/transformers/issues/21961)).

**Describe alternatives you've considered**
Could this possibly be achieved with the use of a `llama_cpp.LogitsProcessor`? I am less familiar with this library and haven't found examples in a similar direction, so am unsure how straightforward this could be to implement using one of those.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.