[Feature Request] OpenAI-compatible `stop` param
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
Using the latest official Docker image, `openmmlab/lmdeploy:v0.4.2`, I served a Llama 2 model, and sent a request with the `stop` parameter of the `/v1/completions` endpoint set to `["\n\n"]`. But the generation didn't stop at a double newline. It generated lots of paragraphs with double newlines between them and kept going until it reached the maximum generation length.
I then saw in the docs that the `stop` param "Only accepts stop words that are encoded to one token index."
Not being able to stop at something simple like "\n\n" in a Llama 2 model is a pretty serious flaw that makes it hard to use this in a production setting. It would be great if the stop param were compatible with OpenAI.
(Also, while I'm here, it would also be very useful to have a `include_stop_str_in_output` option like in vLLM.)
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.