InternLM / InternLM/lmdeploy

[Feature] 目前无法有效的获得停止词的stop_reason,需要获得停止词stop_reason功能

Open
#3,131 8 comments 0 reactions 1 assignee Claimed by @CUHKSZzxy View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

获得停止词可以更好的对一些特殊状态进行处理。因此需要得知停止时的停止词。
vllm 以及openaiapi 的方式都有这个功能,希望能支持

### Related resources

```python
import os
os.environ["LMDEPLOY_USE_MODELSCOPE"] = "True"
from modelscope import snapshot_download
from modelscope.hub.file_download import model_file_download

# MODEL_PATH = snapshot_download('qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4')
MODEL_PATH = snapshot_download('qwen/Qwen2.5-Coder-0.5B-Instruct-AWQ')
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig, PytorchEngineConfig
pipe = pipeline(MODEL_PATH,
backend_config=TurbomindEngineConfig(
# backend_config=PytorchEngineConfig(
max_batch_size=32,
enable_prefix_caching=True,
cache_max_entry_count=0.8,
session_len=8192,
# quant_policy=4,
))
gen_config = GenerationConfig(top_p=0.8,
top_k=40,
temperature=0.8,
do_sample=True,
max_new_tokens=1024,
stop_words=[","],
)
response = pipe(['Hi, pls intro yourself', 'Shanghai is'],
gen_config=gen_config)

print(response)
```
这些代码将得到
```
2025-02-11 16:49:07,073 - lmdeploy - WARNING - tokenizer.py:175 - There are too many(>5) possible indexes may decoding ,, we will use [11] only
[Response(text="Hello! I'm a computer program", generate_token_len=7, input_token_len=24, finish_reason='stop', token_ids=[9707, 0, 358, 2776, 264, 6366, 2025], logprobs=None, logits=None, last_hidden_state=None, index=0), Response(text='Shanghai is a major city in China', generate_token_len=8, input_token_len=22, finish_reason='stop', token_ids=[2016, 30070, 374, 264, 3598, 3283, 304, 5616], logprobs=None, logits=None, last_hidden_state=None, index=1)]
```
无法获知停止原因的具体停止词,另外这个警告怎么解决?
@lvhan028

### Additional context
vllm 通过下列方式可以得知停止词
```
llm_results = [vllm_adapt_result(output) for output in request_outputs]
def vllm_adapt_result(llm_output):
generated_text = llm_output.outputs[0].text
finish_reason = llm_output.outputs[0].finish_reason
stop_reason = llm_output.outputs[0].stop_reason
return (generated_text, finish_reason, stop_reason)
```
_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.