Why are the results worse when using HuggingFace model implementations vs. local?

Nobody has claimed this yet.

Hi, I have a very basic question:

I would like to download Vicuna from HuggingFace (e.g. [this model](https://huggingface.co/lmsys/vicuna-7b-v1.3)) and use it to ask arbitrary questions, like "What topics are discussed in this text?" or "Summarize what happened in this text." If I try this in the [GUI](https://chat.lmsys.org/) I get reasonable answers to all of my questions. I have also tried downloading the weights via FastChat and that works as expected, too.

But, when I try the same thing using HuggingFace versions, the quality is much worse. Rather than respond to the questions as I would expect, it typically just outputs the input text and some additional generated text, which is not what I'm after.

Here is some example code:
```
from transformers import LlamaTokenizer, AutoModelForCausalLM
tokenizer = LlamaTokenizer.from_pretrained("lmsys/vicuna-7b-v1.3", legacy=False)
model = AutoModelForCausalLM.from_pretrained("lmsys/vicuna-7b-v1.3")

input_text = """
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.

USER: In one sentence, describe what happened in this video transcript: Hey everyone, hope you're having a nice day. I'm looking forward to playing piano later, but how are you all? Ha ha, that's nice. Ok, let's get started. This is one of my favorite songs. Oh yeah that sounds fun.

ASSISTANT:
"""

input_ids = tokenizer(input_text, return_tensors="pt")
out = model.generate(input_ids['input_ids'], max_new_tokens=100)
result = tokenizer.decode(out[0])

print(result)
" A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. User: In one sentence, describe what happened in this video transcript: Hey everyone, hope you're having a nice day. I'm looking forward to playing piano later, but how are you all? Ha ha, that's nice. Ok, let's get started. This is one of my favorite songs. Oh yeah that sounds fun. I'm gonna play it now. Wow, that was really good. I'm so happy. I love playing music. I'm so glad I have this channel. I'm so grateful for all of you. I'm so happy to be alive. I love you all."
```

If anyone can point me in the right direction/offer advice I would be super grateful. I feel like I must be missing something obvious. Thank you!

Start with the provided Transformers loading and generation snippet, then compare its prompt and output handling with the FastChat download path described in the issue. Done means identifying why the HuggingFace implementation produces substantially worse answers and documenting or correcting the behavior so the same question receives a useful response.

Written by the indexing model from the issue text.