Terrible Loss when training on SSD MobileNet v2 320x320

Hi, I'm trying to train using ssd mobilenet v2 320x320 using tensorflow 2 and a datatset consisting of 2.4k train and 600 test images. I've been struggling with finding a suitable setting to train on the aforementioned model. I tried modifying the learning_rate_base from its original .8 (which was giving me insane loss values), to 0.03, 0.2, 8e-2 and now 0.004. Pretty much the same occurrence happens as the image below:

Aside of learning rate, I changed warmup_learning_rate to 0.001, and use_dropout to true from convolutional_box_predictor.

Anyone has any idea why this is happening? Should I let it train for longer maybe it stabilises?

Thanks