After QLoRA finetuning, the inference output repeat the instruction over and over again

Nobody has claimed this yet.

I fine-tuned an LLaMA 2 -13 B with QloRA, it was successfully trained and saved without any problem. However, when it comes to the inference part, the output repeats the instructions and makes the output useless. Some of the instructions from Alapca can generate useful output but most do not. Did anyone encounter the same problems and are willing to share their solutions?

My training command is shown as follows:
`deepspeed fastchat/train/train_lora.py \
--model_name_or_path NousResearch/Llama-2-13b-hf \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--data_path ./data/test_fs.json \
--bf16 True \
--output_dir ./test_fs_ql_13 \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 100 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--q_lora True \
--flash_attn False \
--deepspeed playground/deepspeed_config_s2.json \
`

Start by reproducing the reported behavior from the supplied deepspeed fastchat/train/train_lora.py command, focusing on the QLoRA training configuration for NousResearch/Llama-2-13b-hf. The issue does not name an inference file or command, so identify that missing path before comparing the saved model's inference setup; done means generated output no longer repeats the instruction.

Written by the indexing model from the issue text.