The loss is abnormal when fine-tuning meta-llama/Llama-2-7b-hf

Nobody has claimed this yet.

When fine-tuning meta-llama/Llama-2-7b-hf, the loss is abnormal as shown below, but 13b is normal with similar parameters.

training start script:
deepspeed --include="localhost:4,5,6,7" --master_port 29501 fastchat/train/train.py \
--deepspeed ds_config_zero3_auto.json \
--model_name_or_path "/opt/ml/model/Llama-2-7b-hf" \
--data_path ${TRAINING_DATA} \
--output_dir "/opt/ml/model/vicuna-2-7b" \
--num_train_epochs 3 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 16 \
--evaluation_strategy "steps" \
--eval_steps 300 \
--save_strategy "steps" \
--save_steps 300 \
--save_total_limit 6 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.04 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--cache_dir "/tmp" \
--fp16_full_eval \
--fp16 \
--model_max_length 4096 \
--gradient_checkpointing True \
--lazy_preprocess True \

training logs:
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 327.7368, 'learning_rate': 0, 'epoch': 0.0}
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 63.195, 'learning_rate': 0, 'epoch': 0.0}
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 185.4122, 'learning_rate': 0, 'epoch': 0.0}
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 163.9574, 'learning_rate': 0, 'epoch': 0.0}
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 170.4039, 'learning_rate': 0, 'epoch': 0.01}
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
tried to get lr value before scheduler/optimizer started stepping, returning lr=0
{'loss': 120.1931, 'learning_rate': 0, 'epoch': 0.01}

Start with fastchat/train/train.py and the supplied DeepSpeed command, comparing the 7b and 13b runs described in the issue. Inspect the training logs around the repeated learning_rate=0 messages and abnormal loss values. Done means identifying why the 7b run differs and documenting or correcting the cause with a reproducible validation.

Written by the indexing model from the issue text.