InternLM / InternLM/lmdeploy

用lmdeploy部署internlm2_5-7B-chat请求返回为空

Open
#1,991 6 comments 0 reactions 1 assignee Claimed by @lvhan028 View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### 📚 The doc issue

### Code部分:
```
def generate_QA():
server_addr = "..."
client = APIClient(server_addr, api_key=None)
model_name = client.available_models[0]

req = "你是一位善于总结文本大纲的机器人,你的任务是按照以下步骤分析长文本内容,并按照要求进行输出......"

prompt = [{"role": "user", "content": req}]

top_p = 0.9
temperature = 0.7
output_seqlen = 4096
stream_output = False

for response in client.chat_completions_v1(model=model_name, messages=prompt, temperature=temperature, top_p=top_p, n=1, max_tokens=output_seqlen, stream=stream_output, ignore_eos=False):
print(response)
output = response["choices"][0]['message']['content']

return output

print(generate_QA())
```

如果我的prompt是比较简单的问题。比如“介绍一下自己”,返回结果就不为空,但如果prompt较为复杂,返回结果就是空值。如下图:
![image](https://github.com/InternLM/lmdeploy/assets/120365110/ff3bd489-542c-4a03-a496-36c4622c3ceb)
image
![image](https://github.com/InternLM/lmdeploy/assets/120365110/7f0585e5-3028-4765-b55e-228630144c28)

### Suggest a potential alternative/fix

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.