[arcface torch] Loss and learning rate

As it is designed, learning rate raises up linearly in the warmup stage and decreases exponentially in training stage.
During warmup stage, the loss would first decrease before it increases with the growing of learning rate.
The optimal loss pattern is that it will reach minima just at the time when learning rate finishes warming up.
However, recently I found the loss bouncing back so quick when I transferred the training from a 8gpu machine to a 4gpu machine (the sum cuda memory is approximately same).
Is this unexpected loss pattern related to number of GPUs?
Are there any parameters I could adjust?