lm-sys / lm-sys/FastChat

Text generation early stop problem with Vicuna 33B v1.3

Open
#1,821 4 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Hi, I have found a case where 33b stops generating its response.

To reproduce:
1. Go to https://chat.lmsys.org/
2. Select vicuna-33b
3. Say "Hi! How are you?"

The response will be "Hello! As an AI language model, I don" and it will fail on the apostrophe. This problem does not occur with vicuna-13b. I've also been able to reproduce this issue locally in text-generation-webui with different parameters.

This was the first query I tried. It's too bad, I was really excited to use vicuna-33b, but I can't rely on a model which fails (seemingly) so frequently.

Edit: The problem does not exist with the preview version of vicuna-33b

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the behavior at https://chat.lmsys.org/ with vicuna-33b using the reported prompt, then compare it with vicuna-13b and the preview vicuna-33b version. Trace the serving or generation entry point for the selected model and verify that responses no longer stop at the apostrophe without breaking the existing model variants.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.