Fine tuning the flan-t5-large model gives garbage output

Nobody has claimed this yet.

I was able to run the fine tuning script for the flan-t5-large model on a V100 and save the results without issues. Training was done with the example dummy conversations file with this command:

```torchrun --nnodes=1 --nproc_per_node=1 fastchat/train/train_flant5.py \
--model_name_or_path google/flan-t5-large \
--data_path data/dummy_conversation.json \
--bf16 False \
--output_dir ./checkpoints_flant5_large_dummy \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 4 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 300 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap T5Block \
--tf32 False \
--model_max_length 2048 \
--report_to none \
--preprocessed_path FastChat/large_dummy.json \
--gradient_checkpointing True
```

Then I loaded the new model with this command: `python3 -m fastchat.serve.cli --model-path checkpoints_flant5_large_dummy/` and got this when I tried to interact:

```
Human: what is your name?
Assistant: Yes,mètres I’mbling amédias languageuniversal APIlungul languagechemical modelwählt....
```

I was able to verify reasonable responses when using the original google/flan-t5-large model so the environment is likely ok. I am probably running this incorrectly but cannot find any further documentation. There are some answered questions about Vicuna, but not much for Flan-T5. Does anyone know what might be wrong? Either way, it might good to have a little more documentation about the end-to-end fine tuning process for the Flan-F5 models.

Start with fastchat/train/train_flant5.py and the supplied data/dummy_conversation.json command, then compare the fine-tuned checkpoint with the original google/flan-t5-large through fastchat.serve.cli. Check the end-to-end training and loading steps described in the issue. Done means identifying why the checkpoint produces corrupted output and documenting or correcting the reproducible Flan-T5 workflow.

Written by the indexing model from the issue text.