InternLM / InternLM/lmdeploy

[Feature Request] OpenAI-compatible `stop` param

Open
#1,731 9 comments 1 reaction 1 assignee Claimed by @irexyc View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

Using the latest official Docker image, `openmmlab/lmdeploy:v0.4.2`, I served a Llama 2 model, and sent a request with the `stop` parameter of the `/v1/completions` endpoint set to `["\n\n"]`. But the generation didn't stop at a double newline. It generated lots of paragraphs with double newlines between them and kept going until it reached the maximum generation length.

I then saw in the docs that the `stop` param "Only accepts stop words that are encoded to one token index."

Not being able to stop at something simple like "\n\n" in a Llama 2 model is a pretty serious flaw that makes it hard to use this in a production setting. It would be great if the stop param were compatible with OpenAI.

(Also, while I'm here, it would also be very useful to have a `include_stop_str_in_output` option like in vLLM.)

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.