generating result mismatch while using official repo chat and model_worker
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
The model has been trained with SFT using Lora under a fine-tuned chat model which base model is Baichuan-13B-Base.
In multi-chat mode, If use `Baichuan-13B-Chat`'s `model.chat` to generate latest round result, it get right result. But If use model_worker to generate result, it will output the previous round similar result but not related to the latest question.
for example:
Invalid generated result from `model_worker`
request
```
[
{
"role":"user",
"content":"股东除查阅xxx?"
},
{
"role":"assistant",
"content":"根据《公司法》xxxxx, 股东无权进行拍照或摘抄会计凭证的行为。"
},
{
"role":"user",
"content":"1"
}
]
```
response
```
{
"role":"assistant",
"content":"根据《公司法》第xxxx,股东无权进行拍照或摘抄会计凭证的行为。"
}
```
it will generate similar result with first answer.
valid result from `Baichuan official repo`
response
```
{
"role":"assistant",
"content":"您好,我是一个 AI 助手,很高兴为您提供帮助。请问有什么我可以为您做的?"
}
```
The main difference between them is `Baichuan official repo` will add a `eos token` after each round. So I add `eos token` in the template and make it decode as `[2]`(it will also decode as `[2154, 31114, 31219]`). And then, it got a valid result.
Do you have any suggestions?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the multi-chat request through model_worker and compare its prompt construction with Baichuan-13B-Chat's model.chat behavior. Inspect the model_worker chat template and EOS-token handling described in the issue; done means the latest user turn produces a response related to that turn rather than repeating the previous answer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100