Potential Issue of Vicuna v1.5 Training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
Thanks for the great work! Recently, while evaluating Vicuna-v1.5, we found that Vicuna-v1.5k tends to generate some strange artifacts ("\\_") when asked to create code (see example below).
And we verified that this issue mostly happens in the Vicuna model, and this issue is more severe in the 13B model:
We did a quick check on the [publicly available ShareGPT data](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered), we found that ~15% of their conversations contain this "\\_" artifact.
Feel free to check Appendix D.1 of our [preprint](https://arxiv.org/abs/2309.10691) for more details :)
I know Vicuna is fine-tuned on ShareGPT data, and Is it possible that the training data Vicuna used also contains such an artifact, which causes Vicuna to exhibit such unwanted behavior?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked ShareGPT dataset and the Vicuna-v1.5 training context described in the issue. Reproduce the artifact with code-generation prompts and compare affected conversations with the training data. Done means establishing whether the artifact comes from the data and documenting the evidence or a mitigation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100