Loss reaches 0 when finetuning 7B model using 2xA100 80G

I'm using the config below and I load the base model as torch.float16

--model_name_or_path llama_model
--data_path data.json
--bf16 True
--num_train_epochs $3
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--gradient_accumulation_steps 16
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 1200
--save_total_limit 3
--learning_rate 2e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--model_max_length 2048
--fsdp "full_shard auto_wrap"
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
--tf32 True
--gradient_checkpointing True
--lazy_preprocess True
--report_to tensorboard

The report names data.json, a Llama model, mixed-precision settings, and FSDP across two A100 GPUs. Start by reproducing the training run with the supplied configuration and inspect the loss behavior and training logs. Done means identifying why the loss reaches 0 and verifying the behavior with a corrected configuration.