Loss nan problem

I used arcface_torch to train with my own dataset as follow:
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup python -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py --network r50 --loss arcface
But it encountered the loss NAN problem as follow:
Training: 2021-06-01 16:22:55,751-Speed 219.17 samples/sec Loss 2.3655 Epoch: 0 Global Step: 14550 Required: 102 hours
Training: 2021-06-01 16:23:39,606-Speed 218.91 samples/sec Loss 2.4382 Epoch: 0 Global Step: 14600 Required: 102 hours
Training: 2021-06-01 16:24:23,457-Speed 218.93 samples/sec Loss 2.4757 Epoch: 0 Global Step: 14650 Required: 102 hours
Training: 2021-06-01 16:25:07,307-Speed 218.93 samples/sec Loss 2.4868 Epoch: 0 Global Step: 14700 Required: 102 hours
Training: 2021-06-01 16:25:51,183-Speed 218.82 samples/sec Loss nan Epoch: 0 Global Step: 14750 Required: 102 hours
Training: 2021-06-01 16:26:33,973-Speed 224.36 samples/sec Loss nan Epoch: 0 Global Step: 14800 Required: 102 hours
Training: 2021-06-01 16:27:16,771-Speed 224.32 samples/sec Loss nan Epoch: 0 Global Step: 14850 Required: 102 hours
Training: 2021-06-01 16:27:59,556-Speed 224.38 samples/sec Loss nan Epoch: 0 Global Step: 14900 Required: 102 hours
Training: 2021-06-01 16:28:42,450-Speed 223.81 samples/sec Loss nan Epoch: 0 Global Step: 14950 Required: 102 hours

Someone said that reducing LR can avoid this mistake.
but even I reduced the learning rate to 0.001, it still happens.

Start with the train.py entry point and the provided torch.distributed.launch command, then reproduce the run on the reported dataset and inspect the loss around global step 14750. Compare behavior with the reported learning rates and identify the cause of the NaN loss; done means training completes without NaN loss and the cause is documented.

Written by the indexing model from the issue text.