Model only manages to converge when not using loss scaling

So I've run into a very weird problem regarding the usage of two models with amp.

I have two models model A and model B.
Model A is being loaded from a checkpoint that was pretrained normally using amp (O1) and will be frozen throughout training.
Model B is initialized and will be trained using amp.
We've trained different setups from these options:

# initialization
1. model_B, optimizer_B = amp.initialize(model_B, optimizer_B, opt_level='O1')
2. model_A = amp.initialize(model_A, opt_level='O1')
3. [model_A, model_B], optimizer_B = amp.initialize([model_A, model_B], optimizer_B, opt_level='O1')
4. [model_A, model_B], optimizer_B = amp.initialize([model_A, model_B], optimizer_B, opt_level='O1', loss_scale=x)

# backprop
1. losses.backward()
2. with amp.scale_loss(losees, optimizer_B) as scaled_losses:
        scaled_losses.backward()

When does the training works and the model DOES manage to converge:

1. When using initialization number 2 (not initializing model B) and with backprop number 1. (In this case model_B still is being cast to FP16 for some reason).
2. When using initialization number 3 with backprop number 1.
3. When using initialization number 4 using any loss_scale value with backprop number 1 (because loss scaling isn't used)

When does the model NOT CONVERGE:

1. When scaling the loss with scale 1 (backprop number 2), which is supposed to be equal to not scaling at all, but still somehow doesn't converge.
2. When using any initialization with backprop number 2
3. When using initialization number 2 (not initializing B) and with backprop number 1 and when running model_B on some input we wrap that with amp.disable_casts() so that model_B stays always as FP32.
4. When removing every mention of amp and training with pytorch only.

We have no idea why the model works on mixed precision but doesn't when training only using normal pytorch and we'd love any advice.

No files or tests are named. Start by reproducing the initialization and backpropagation combinations around amp.initialize, amp.scale_loss, and amp.disable_casts, comparing scaled and unscaled behavior in PyTorch. Done means identifying the cause of the convergence difference and documenting or fixing it with a reproducible test.