Reproducibility of mixed precision training.

When I training with opt 1, I failed to reproduce the training loss and etc.
I used pytorch 1.6.0 and CUDA 10.2; it works well with opt 0 since I have fixed all random seed according pytorch Docs.
I found that the difference between two runs is minor, some output are presented below, and I have not used scale loss.

run1

0/36000 [00:00<?, ?it/s]00001, da_loss 579.81177, id_loss 35.67139, age_loss 614.51257, loss 37.44553, lr 0.00010, ^M  0%|          | 1/36000 [00:00<7:50:07,  1.28it/s]
00002, da_loss 534.40143, id_loss 35.92854, age_loss 568.46362, loss 37.56580, lr 0.00020, ^M  0%|          | 2/36000 [00:01<6:42:24,  1.49it/s]
00003, da_loss 551.62842, id_loss 35.78385, age_loss 579.52393, loss 37.46664, lr 0.00030,
^M  0%|          | 3/36000 [00:01<5:55:39,  1.69it/s]^M  0%|          | 4/36000 [00:02<5:22:08,  1.86it/s]00004, da_loss 609.66608, id_loss 35.81017, age_loss 646.82190, loss 37.67632, lr 0.00040,
00005, da_loss 583.82153, id_loss 35.79406, age_loss 617.21631, loss 37.57892, lr 0.00050,
^M  0%|          | 5/36000 [00:02<4:58:54,  2.01it/s]00006, da_loss 570.41064, id_loss 35.66693, age_loss 604.19055, loss 37.41194, lr 0.00060, ^M  0%|          | 6/36000 [00:02<4:42:30,  2.12it/s]
00007, da_loss 584.26428, id_loss 35.71205, age_loss 619.66589, loss 37.50025, lr 0.00070,
^M  0%|          | 7/36000 [00:03<4:31:00,  2.21it/s]00008, da_loss 542.56897, id_loss 35.88947, age_loss 574.64789, loss 37.54926, lr 0.00080,
^M  0%|          | 8/36000 [00:03<4:23:10,  2.28it/s]00009, da_loss 598.12805, id_loss 35.56038, age_loss 633.80774, loss 37.39044, lr 0.00090,

run1

^M  0%|          | 0/36000 [00:00<?, ?it/s]00001, da_loss 579.81177, id_loss 35.67139, age_loss 614.51257, loss 37.44553, lr 0.00010,
^M  0%|          | 1/36000 [00:00<8:57:31,  1.12it/s]00002, da_loss 534.39661, id_loss 35.92825, age_loss 568.47205, loss 37.56551, lr 0.00020,
^M  0%|          | 2/36000 [00:01<7:30:09,  1.33it/s]00003, da_loss 551.63147, id_loss 35.78336, age_loss 579.51654, loss 37.46613, lr 0.00030,
^M  0%|          | 3/36000 [00:01<6:27:45,  1.55it/s]^M  0%|          | 4/36000 [00:02<5:44:11,  1.74it/s]00004, da_loss 609.66547, id_loss 35.81051, age_loss 646.82452, loss 37.67667, lr 0.00040,
00005, da_loss 583.82300, id_loss 35.79415, age_loss 617.21686, loss 37.57901, lr 0.00050, ^M  0%|          | 5/36000 [00:02<5:13:50,  1.91it/s]
00006, da_loss 570.41791, id_loss 35.66760, age_loss 604.19031, loss 37.41263, lr 0.00060, ^M  0%|          | 6/36000 [00:02<4:52:54,  2.05it/s]
^M  0%|          | 7/36000 [00:03<4:37:59,  2.16it/s]00007, da_loss 584.26135, id_loss 35.71220, age_loss 619.66119, loss 37.50039, lr 0.00070,
00008, da_loss 542.57513, id_loss 35.89023, age_loss 574.64459, loss 37.55003, lr 0.00080,
^M  0%|          | 8/36000 [00:03<4:27:43,  2.24it/s]00009, da_loss 598.11682, id_loss 35.56154, age_loss 633.79926, loss 37.39157, lr 0.00090,

I wonder what happens that result in this minor difference?

Reproduce the two runs with PyTorch 1.6.0 and CUDA 10.2, using the fixed-seed setup and comparing mixed-precision opt 1 with opt 0. Start with the linked PyTorch reproducibility guidance, then identify the first differing outputs and determine whether the variation is expected; done means documenting the cause or a reproducible corrective path.