deepseek-ai / deepseek-ai/DeepSeek-Coder

使用vllm加速inference后输出容易不符合格式要求

Open
#158 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
24.3k
Forks
2.9k
PR merge metrics
No merged PRs in 30d

Description

我在prompt中要求模型仅输出以def execute_command为开头的python函数,在原生的模型中测试没有问题,但是在利用vllm加速之后,容易输出类似如图
iShot_2024-04-25_18 18 59
不符合格式要求的输出。
以及也有可能输出对函数为什么要这么写的解释,但我不需要这些额外的输出。
请问有什么办法抑制这样的输出,使其符合我的格式要求吗?

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is identified in the issue. First reproduce the same prompt with native inference and vLLM, then compare the generated output and available generation settings; done means establishing a documented or tested way to keep the response limited to the requested Python function.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.