[BUG] Context parallel `nan` loss

**Describe the bug**

When enabling context parallel and when all the tokens on a single GPU is masked out, the loss and grad becomes NaN.

see also: https://github.com/NVIDIA/NeMo/issues/14488

**To Reproduce**
Use CP > 2 and pad a short training data 2x it's original length

**Expected behavior**
model trains.

**Environment (please complete the following information):**
- Megatron-LM commit ID version 0.31.1
- PyTorch version
- CUDA version
- NCCL version

**Proposed fix**
I've identified the issue. submitting a PR soon