Llama-2 loss and learning rate is always 0 after first step

Nobody has claimed this yet.

The log appears like this:

{'loss': 1.8709, 'learning_rate': 0.0, 'epoch': 0.0}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.01}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.01}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.01}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.01}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.02}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.02}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.02}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.02}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.03}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.03}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.03}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.04}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.04}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.04}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.04}

{'loss': 0.0, 'learning_rate': 0.0, 'epoch': 0.05}:

Script:

deepspeed --include="localhost:0,1,2,3" --master_port=20001 fastchat/train/train_mem.py \
--deepspeed playground/deepspeed_config_s6.json \
--model_name_or_path NousResearch/Redmond-Puffin-13B \
--data_path data/dummy_conversation.json \
--output_dir PUFFIN_ON_ZOOTIEZ \
--num_train_epochs 2 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--gradient_accumulation_steps 8 \
--evaluation_strategy epoch \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 10 \
--learning_rate 1e-4 \
--weight_decay 0. \
--warmup_ratio 0.1 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fp16 \
--cache_dir "/tmp" \
--model_max_length 4096 \
--gradient_checkpointing True \
--lazy_preprocess True

Start by reproducing the reported command with fastchat/train/train_mem.py and playground/deepspeed_config_s6.json, using the listed Llama-2 training settings. Inspect the training and logging path around the first optimizer step. Done means the loss and learning-rate logs no longer remain at 0.0 after the first step.

Written by the indexing model from the issue text.