lm-sys / lm-sys/FastChat

generating result mismatch while using official repo chat and model_worker

Open
#2,605 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

The model has been trained with SFT using Lora under a fine-tuned chat model which base model is Baichuan-13B-Base.

In multi-chat mode, If use `Baichuan-13B-Chat`'s `model.chat` to generate latest round result, it get right result. But If use model_worker to generate result, it will output the previous round similar result but not related to the latest question.

for example:
Invalid generated result from `model_worker`
request
```
[
{
"role":"user",
"content":"股东除查阅xxx?"
},
{
"role":"assistant",
"content":"根据《公司法》xxxxx, 股东无权进行拍照或摘抄会计凭证的行为。"
},
{
"role":"user",
"content":"1"
}
]
```

response
```
{
"role":"assistant",
"content":"根据《公司法》第xxxx,股东无权进行拍照或摘抄会计凭证的行为。"
}
```
it will generate similar result with first answer.

valid result from `Baichuan official repo`
response
```
{
"role":"assistant",
"content":"您好,我是一个 AI 助手,很高兴为您提供帮助。请问有什么我可以为您做的?"
}
```

The main difference between them is `Baichuan official repo` will add a `eos token` after each round. So I add `eos token` in the template and make it decode as `[2]`(it will also decode as `[2154, 31114, 31219]`). And then, it got a valid result.

Do you have any suggestions?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the multi-chat request through model_worker and compare its prompt construction with Baichuan-13B-Chat's model.chat behavior. Inspect the model_worker chat template and EOS-token handling described in the issue; done means the latest user turn produces a response related to that turn rather than repeating the previous answer.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.