[ENHANCEMENT] Sequential Deletion of Old Checkpoint Files Slows Down Checkpointing

**Is your feature request related to a problem? Please describe.**
Currently, during checkpointing in Nemo Megatron, the deletion of older checkpoint files is performed sequentially. This significantly slows down the process of deletion phase, as deleting files in parallel could dramatically improve efficiency.

**Describe the solution you'd like**
Optimize checkpoint deletion by enabling parallel execution, allowing multiple old checkpoint files to be deleted simultaneously. This would exponentially improve deletion speed, reducing overall checkpointing time.

**Describe alternatives you've considered**
1. Batching file deletions instead of strict sequential execution.
2. Using multi-threading or async deletion mechanisms.
3. Offloading deletion tasks to a background process to prevent blocking.

**Proposed implementation**
Leverage parallel or asynchronous deletion techniques, such as:

1. Using concurrent workers to delete files in parallel.
2. Utilizing async I/O or thread pools to manage deletion efficiently.
3. Implementing a batched approach to delete multiple files at once.

**Additional context**

- This change would significantly speed up checkpointing, especially for large models with numerous checkpoint files.
- Distributed storage systems like GCS handle parallel deletions efficiently, making this a viable optimization.