Training SAM2 ; "train_all_loss_mask" component of loss function never converge

Hi all I hope you are doing well.

I am trying to retrain SAM2 following the informations from the paper. I used the same batch size etc. You know that the loss is composed of 4 elements with the weights (1, 1, 1, 20). During the training, all loss components converges except "train_all_loss_mask" (the component with the weight 20). I tried multiple using to improve it (add warmup, increase batch size, decrease learning rate, etc.) but nothing works, the loss curve doesnt converge and has multiple spike. Does one of you also encounter ? if yes, how do you resolve that ? I used SA-V train set for the training.

Thank you for your help !