[BUG] NaNs when unfreezing vision encoder in the multi-modal example

**Describe the bug**
In the multimodal example in https://github.com/NVIDIA/Megatron-LM/tree/main/examples/multimodal , wehn we run the scripts for either pretraining or finetuning, if we unfreeze the vision encoder we get immediate NaNs on the first step.
I've tried lowering the learning rate, changing the data, etc... so it appears to not be data/optimization related

**To Reproduce**
Try to run the example scripts in the example, by removing `--freeze-ViT` flag

**Expected behavior**
It is supposed to train without NaNs_

**Stack trace/logs**
If applicable, add the stack trace or logs from the time of the error.

**Environment (please complete the following information):**
- Custom fork (https://github.com/deep-spin/Megatron-LM-pretrain) with minor changes, last commit from upstream: cc207f809cd705a0d6f506e748405d23c28c1a92
- PyTorch version: 2.5.1+cu124
- CUDA version: 12.4
- NCCL version> V12.4.99