Loss could not converge when training from scratch

My experiment is lauched with information
- GPU: 2
- Model: IResNet50
- Pretrained: No
- Marginal softmax: ArcFace (s=30, m=0.5), do not use partial_fc
- Batchsize : 64*2=128
- Warm up: 0
- Optimizer: AdamW
- Init learning rate: 1e-1
- Loss function: cross entropy
- Dataset: MS1Mv2
But I am facing to some weird cases
- Model could not converge and loss function is about 16 in many epochs (It does not like log of Insightface repo, loss seems converge so fast)
Thank you for your attention