[ENHANCEMENT] Nemo Megatron Retries Missing Index Files Without Exponential Backoff

**Is your feature request related to a problem? Please describe.**
During training, Nemo Megatron looks for index files from different nodes. When a file is not found, the system retries, but the retries do not follow an exponential backoff pattern. This results in erratic and frequent server calls.

**Describe the solution you'd like**
Implement an exponential backoff strategy for retrying missing index file lookups. This would:
1. Reduce server load by spacing out retries progressively.
2. Improve the chances of successful file retrieval.
3. Prevent unnecessary resource consumption due to aggressive retries.

**Describe alternatives you've considered**
1. Implementing a fixed delay instead of exponential backoff (less efficient).
2. Introducing a jitter mechanism to prevent synchronized retry storms from multiple nodes.

**Proposed implementation**
1. Modify the retry mechanism to follow an exponential backoff strategy, with increasing wait times (e.g., 1s, 2s, 4s, etc.).
2. Optionally, add jitter to prevent multiple nodes from retrying at the same time.
3. Implement logging to track and fine-tune the retry intervals.

**Additional context**
Lookup Timestamps from one of the nodes in our test training using Nemo(Megatron) 24.09