When using the FastChat OpenAI API to deploy the Qwen-7B-Chat model with stop configuration, if the custom stop_word is not triggered, the model will not stop promptly until it reaches the max_tokens limit.

Bug Description

When using the FastChat OpenAI API to deploy the Qwen-7B-Chat model with stop configuration, if the custom stop_word is not triggered, the model will not stop promptly until it reaches the max_tokens limit.

Steps to Reproduce

1. python -m fastchat.serve.controller
2. python -m fastchat.serve.model_worker --model-path Qwen-7B-Chat --model-name gpt-3.5-turbo,text-davinci-003,text-embedding-ada-002
3. python -m fastchat.serve.openai_api_server
4. POST /worker_generate_stream

Screenshots/Recordings

• The yellow highlight represents the output I expected, but it didn't stop promptly as desired.

Additional Information

After investigation, I found that FastChat/blob/main/fastchat/serve/openai_api_server.py#L278 , When I add a stop, the default stop will not be configured_ Token_ Ids, this will only cause llm to stop_ Stop after Word appears. In theory, llm should also be stopped at the end of the answer, which is to say, stop_ Token_ Automatically stop when the token corresponding to ids appears.

At present, I have two ways to solve this situation, but I am not sure if it will have an impact on other LLMs

1. https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/inference.py#L185 Set skip_special_tokens=False and add “<|im_end|>” to stop_word
2. https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/openai_api_server.py#L278 When there is a stop, also update "stop_token_ids": conv.stop_token_ids

Reproduce the issue with the listed controller, model_worker, and openai_api_server commands, then inspect fastchat/serve/openai_api_server.py around line 278 and fastchat/serve/inference.py around line 185. Compare custom stop handling with conv.stop_token_ids and skip_special_tokens behavior for Qwen-7B-Chat and other LLMs. Done means generation stops at the model's answer-ending token when the custom stop word is absent without breaking other models.