arcface_torch configs/ms1mv3_r18 -----> Loss nan

**when i train arcface_torch configs/ms1mv3_r18, have trouble for Loss nan**

Here's train log!

(pytorch1.6) admin0:~/deepinsight/insightface/recognition/arcface_torch$ srun -N 1 --gres=gpu:8 -p algorithm-chejian python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" --master_port=1234 train.py configs/ms1mv3_r18
srun: job 102171 queued and waiting for resources
srun: job 102171 has been allocated resources
Training: 2021-07-16 20:56:48,645-rank_id: 0
Training: 2021-07-16 20:57:00,703-softmax weight init successfully!
Training: 2021-07-16 20:57:00,703-softmax weight mom init successfully!
Training: 2021-07-16 20:57:00,704-: loss arcface
Training: 2021-07-16 20:57:00,704-: network r18
Training: 2021-07-16 20:57:00,704-: resume False
Training: 2021-07-16 20:57:00,704-: output work_dirs/ms1mv3_r18
Training: 2021-07-16 20:57:00,705-: dataset ms1m-retinaface-t1
Training: 2021-07-16 20:57:00,705-: embedding_size 512
Training: 2021-07-16 20:57:00,705-: sample_rate 1.0
Training: 2021-07-16 20:57:00,705-: fp16 True
Training: 2021-07-16 20:57:00,705-: momentum 0.9
Training: 2021-07-16 20:57:00,705-: weight_decay 0.0005
Training: 2021-07-16 20:57:00,705-: batch_size 384
Training: 2021-07-16 20:57:00,705-: lr 0.1
Training: 2021-07-16 20:57:00,705-: rec /lustre/users/chenguang/data/face/faces_emore
Training: 2021-07-16 20:57:00,705-: num_classes 93431
Training: 2021-07-16 20:57:00,705-: num_image 5179510
Training: 2021-07-16 20:57:00,705-: num_epoch 25
Training: 2021-07-16 20:57:00,705-: warmup_epoch -1
Training: 2021-07-16 20:57:00,706-: decay_epoch [10, 16, 22]
Training: 2021-07-16 20:57:00,706-: val_targets ['lfw', 'cfp_fp', 'agedb_30']
Training: 2021-07-16 20:57:00,706-: resume_model backbone.pth
Training: 2021-07-16 20:57:00,706-: warmup_step -1895
Training: 2021-07-16 20:57:00,706-: total_step 47375
Training: 2021-07-16 20:57:00,706-: decay_step [18953, 30326, 41698]
...
...
Training: 2021-07-16 21:11:10,631-Speed 2243.92 samples/sec Loss 25.8109 LearningRate 0.6000 Epoch: 1 Global Step: 2050 Fp16 Grad Scale: 3072 Required: 5 hours
Training: 2021-07-16 21:11:25,216-Speed 10532.00 samples/sec Loss 25.8466 LearningRate 0.6000 Epoch: 1 Global Step: 2100 Fp16 Grad Scale: 3072 Required: 5 hours
Training: 2021-07-16 21:11:40,205-Speed 10247.92 samples/sec Loss 25.9128 LearningRate 0.6000 Epoch: 1 Global Step: 2150 Fp16 Grad Scale: 6144 Required: 5 hours
Training: 2021-07-16 21:11:56,593-Speed 9373.46 samples/sec Loss 26.4034 LearningRate 0.6000 Epoch: 1 Global Step: 2200 Fp16 Grad Scale: 6144 Required: 5 hours
Training: 2021-07-16 21:12:13,489-Speed 9091.42 samples/sec Loss 26.0906 LearningRate 0.6000 Epoch: 1 Global Step: 2250 Fp16 Grad Scale: 12288 Required: 5 hours
Training: 2021-07-16 21:12:31,352-Speed 8598.87 samples/sec Loss 25.9126 LearningRate 0.6000 Epoch: 1 Global Step: 2300 Fp16 Grad Scale: 3072 Required: 5 hours
Training: 2021-07-16 21:12:48,455-Speed 8981.49 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2350 Fp16 Grad Scale: 1 Required: 5 hours
Training: 2021-07-16 21:13:03,711-Speed 10068.48 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2400 Fp16 Grad Scale: 0 Required: 5 hours
Training: 2021-07-16 21:13:19,859-Speed 9512.47 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2450 Fp16 Grad Scale: 0 Required: 5 hours
Training: 2021-07-16 21:13:35,858-Speed 9600.90 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2500 Fp16 Grad Scale: 0 Required: 5 hours
Training: 2021-07-16 21:13:52,046-Speed 9488.67 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2550 Fp16 Grad Scale: 0 Required: 5 hours
Training: 2021-07-16 21:14:08,330-Speed 9432.86 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2600 Fp16 Grad Scale: 0 Required: 5 hours
Training: 2021-07-16 21:14:24,139-Speed 9716.21 samples/sec Loss nan LearningRate 0.6000 Epoch: 1 Global Step: 2650 Fp16 Grad Scale: 0 Required: 5 hours