Conversation templates Api

Nobody has claimed this yet.

Hi!
I would like to know if the conversation templates can be changed. I am using two models T5 and vicuna-7b. The default T5 conversation template is long and reduces the input size of the context (2048) in the tokenization. So, with T5 I managed to remove the conversation template in the huggingface_api.py just passing the context + question as a message. Like this:

def main(args, msg, model, tokenizer):
#conv = get_conversation_template(args.model_path)
#conv.append_message(conv.roles[0], msg)
#conv.append_message(conv.roles[1], None)
#prompt = conv.get_prompt()
# input_ids = tokenizer([prompt]).input_ids

input_ids = tokenizer([msg]).input_ids

output_ids = model.generate(
torch.as_tensor(input_ids).cuda(),
do_sample=True,
temperature=args.temperature,
max_new_tokens=args.max_new_tokens,
)
if model.config.is_encoder_decoder:
output_ids = output_ids[0]
else:
output_ids = output_ids[0][len(input_ids[0]) :]
outputs = tokenizer.decode(
output_ids, skip_special_tokens=True, spaces_between_special_tokens=False
)

return outputs

However, If I do the same but using the weights from vicuna-7B the outputs are always "." . I tried to use the conversation template for Vicuna and it works. I also tried to put the system message = "" and it also works. So, why is the conversation template necessary for vicuna 7-B but not for T5? What happens if I change the conversation template for inference and fine-tuning?

Start in huggingface_api.py at main and compare the commented conversation-template path with tokenizer([msg]). Reproduce the T5 and Vicuna behavior, then inspect how the selected template formats prompts for inference and fine-tuning. Done means documenting why the models differ and what template changes are safe.

Written by the indexing model from the issue text.