abetlen / abetlen/llama-cpp-python
Capture the stop token upon termination
Open
- Dominant language
- Python
- Stars
- 10.6k
- Forks
- 1.4k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
I have a list of stop tokens `["FOO", "BAR"]` and I would like to know which one triggered the halting mechanism.
**Describe the solution you'd like**
I want to see the corresponding token in the response object, on top of `reason: stop`/
**Describe alternatives you've considered**
Until now I have to increment max_tokens incrementally while the stop token is not spotted in the response.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.