[BUG] Multiple Nodes Attempt to Create Checkpoint Folder Simultaneously, Causing Errors

**Describe the bug**
During checkpointing in Nemo Megatron, all nodes attempt to create the checkpoint folder simultaneously. Since only one of them succeeds, multiple nodes encounter "create folder" errors. This results in unnecessary error messages in logs and observability graphs, leading to confusion and wasted resources.

**To Reproduce**
Steps to reproduce the behavior:

1. Run Nemo Megatron with checkpointing enabled in a multi-node setup.
2. Observe that all nodes try to create the checkpoint folder at the same time.
3. Only one succeeds, while others fail with folder creation errors.
4. Create failure errors appear in logs and monitoring dashboards

**Expected behavior**
Only a single node should attempt to create the checkpoint folder, while others should wait for its creation before proceeding.

**Stack trace/logs**
Sample folder name: "nemo-experiments/test-run/checkpoints/megatron_llama--val_loss=0.06-step=30-consumed_samples=61440.0-last/"

**Environment (please complete the following information):**

Framework: NeMo 24.09 (Megatron)
Model: Llama-3.1-70B
Megatron Core: 0.90, FSDP where DP=1.
Checkpoint used: distributed (.distcp)
Environment: GKE cluster with Cloud Storage Fuse CSI driver enabled (cluster version - 1.31.1-gke.1846000), Using local File API writing to GCSFuse, which remaps the local file API to GCS calls.
Number of Nodes: 128
Node Type: a3-megagpu-8g
GCS Bucket Type: Hierarchical Namespace GCS Buckets.
Checkpoint Size: Each checkpoint write is 1.3TB written from a set of nodes, in parallel, with distributed files with file sizes varying from 200 MB to 1GB