The verification accuracy of arface_torch is not improving

Hello,

I am training a mobilefacenet(embedding=128) with CASIA_Webface using arcface_torch. The loss has decreased from 53.7805 to 2.8381. However, the verification accuracies of lfw, cfp_fp, and agedb_30 remain 0.5, which does not make any sense.
Could you please point out where I might go wrong? Thank you.

----------------------------------------------------------------------------------------------------------------------------------------------
Training: 2021-09-03 04:05:58,634-Speed 322.19 samples/sec Loss 2.6971 LearningRate 0.0500 Epoch: 1 Global Step: 2000 Required: 14 h ours
testing verification..
(12000, 128)
infer time 44.320912999999905
Training: 2021-09-03 04:06:46,999-[lfw][2000]XNorm: 9.465224
Training: 2021-09-03 04:06:46,999-[lfw][2000]Accuracy-Flip: 0.50000+-0.00000
Training: 2021-09-03 04:06:46,999-[lfw][2000]Accuracy-Highest: 0.50000
testing verification..
(14000, 128)
infer time 51.230545000000006
Training: 2021-09-03 04:07:42,615-[cfp_fp][2000]XNorm: 9.465774
Training: 2021-09-03 04:07:42,616-[cfp_fp][2000]Accuracy-Flip: 0.49986+-0.00043
Training: 2021-09-03 04:07:42,616-[cfp_fp][2000]Accuracy-Highest: 0.49986
testing verification..
(12000, 128)
infer time 43.916425999999944
Training: 2021-09-03 04:08:30,471-[agedb_30][2000]XNorm: 9.467841
Training: 2021-09-03 04:08:30,471-[agedb_30][2000]Accuracy-Flip: 0.50033+-0.00067
Training: 2021-09-03 04:08:30,471-[agedb_30][2000]Accuracy-Highest: 0.50033