Terrible Results in T2I Fine-tuining.

After my tests, I discovered a very strange issue.
First, I fine-tuned on my T2I dataset. Below is the script I used for fine-tuning.
```bash
torchrun \
--nnodes=1 \
--node_rank=0 \
--nproc_per_node=8 \
--master_addr=127.0.0.1 \
--master_port=12345 \
train/pretrain_unified_navit.py \
--dataset_config_file ./data/configs/t2i.yaml \
--layer_module Qwen2MoTDecoderLayer \
--max_latent_size 64 \
--model_path $model_path \
--use_flex True \
--resume-from $model_path \
--finetune_from_hf True \
--auto_resume True \
--resume-model-only True \
--finetune-from-ema True \
--num_shard 8 \
--wandb_offline True \
--results_dir $output_path \
--checkpoint_dir $ckpt_path \
--log_every 10 \
--save_every 1000 \
--max_save_num 3 \
--lr 2e-6 \
--total_steps 20000 \
--visual_und False\
--expected_num_tokens 10240 \
--max_num_tokens 11520 \
--max_num_tokens_per_sample 10240 \
--num_workers 1
```
According to the training log, I found that the model experienced a sharp increase in loss around step 5000, which seemed to indicate a severe problem of knowledge forgetting.
The following is a comparison of the quantitative results between the fine-tuned model and the zero-shot model.

Then, since the ckpt saved during training lacked the vit-related components (#160 ), I supplemented the vit components from the official model to the ckpt saved during training based on the script provided by @AbhinavJangra29 (🤗THANKS)
. After my tests, I found that the model with the supplemented vit performed slightly worse in inference results compared to the directly saved model. The quantitative results are as follows:

Start with train/pretrain_unified_navit.py and data/configs/t2i.yaml, reproducing the reported fine-tuning command and examining the loss near step 5000. Compare inference from the directly saved checkpoint with the checkpoint supplemented with ViT components, using the quantitative results in the issue as the baseline. Done means identifying and documenting the cause of the loss increase and checkpoint discrepancy, or confirming a reproducible fix.

Written by the indexing model from the issue text.