DLIO hangs during first epoch when running with certain accelerators counts

Seeing DLIO hang during the first epoch when running certain accelerators counts. Running with 6x a100 or 7x a100 will cause the test to hang after printing the summary of epoch 1 and CTR+C is needed to stop the test. The strange thing is, running with 5x a100 or 8x a100 passes. I see this same behavior with other clients and when running multiple clients. The output below was collected with openmpi 4.1.2 and python 3.10.12 but I've also used openmpi 5.0.8 but still get the same behavior. Is there any additional logging that can be enabled to help narrow down and fix the issue?

5x a100

[OUTPUT] 2025-06-21T15:32:56.403469 Saved outputs in /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_151706 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:204]
[OUTPUT] Averaged metric over all steps/epochs
[METRIC] ==========================================================
[METRIC] Number of Simulated Accelerators: 5
[METRIC] Training Accelerator Utilization [AU] (%): 97.7676 (0.1091)
[METRIC] Training Throughput (samples/second): 53.5577 (0.0599)
[METRIC] Training I/O Throughput (MB/second): 7487.8664 (8.3689)
[METRIC] train_au_meet_expectation: success
[METRIC] ==========================================================
 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:226]
[OUTPUT] 2025-06-21T15:32:56.410701 outputs saved in RANKID_output.json [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:456]
2025-06-21 15:33:27|STATUS: Writing metadata for benchmark to: /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_151706/training_20250621_151706_metadata.json
Setting attr from num_accelerators to 5
Hosts is: ['192.168.20.21']
Hosts is: ['192.168.20.21']

6x a100

[INFO] 2025-06-21T14:39:05.657766 Maximum number of steps reached [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/main.py:350]
[DEBUG] 2025-06-21T14:39:05.667202 Rank 5 returned after 238 steps. [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/main.py:407]
[DEBUG] my_rank: 5, start_sample: 8335, end_sample: 9999 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/config.py:441]
[OUTPUT] 2025-06-21T14:39:05.658318 Ending block 1 - 238 steps completed in 158.11 s [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:338]
[OUTPUT] 2025-06-21T14:39:05.668577 Epoch 1 - Block 1 [Training] Accelerator Utilization [AU] (%): 98.2719 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:340]
[OUTPUT] 2025-06-21T14:39:05.668684 Epoch 1 - Block 1 [Training] Throughput (samples/second): 64.5567 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:341]
[OUTPUT] 2025-06-21T14:39:05.668760 Epoch 1 - Block 1 [Training] Computation time per step (second): 0.6366+/-0.0001 (set value: {'mean': 0.636}) [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:342]
2025-06-21 14:52:48|WARNING: Received signal SIGINT (2)
2025-06-21 14:52:48|INFO: Exiting immediately due to SIGTERM
2025-06-21 14:52:48|ERROR: Error occurred while executing command: INTERRUPTED
2025-06-21 14:52:48|STATUS: Writing metadata for benchmark to: /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_143620/training_20250621_143620_metadata.json
2025-06-21 14:52:48|ERROR: Benchmark failed after 1 iterations
Setting attr from num_accelerators to 6
Hosts is: ['192.168.20.21']
Hosts is: ['192.168.20.21']

7x a100 -

[INFO] 2025-06-21T15:04:00.500743 Maximum number of steps reached [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/main.py:350]
[DEBUG] 2025-06-21T15:04:00.510045 Rank 6 returned after 204 steps. [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/main.py:407]
[DEBUG] my_rank: 6, start_sample: 8574, end_sample: 9999 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/config.py:441]
[OUTPUT] 2025-06-21T15:04:00.501313 Ending block 1 - 204 steps completed in 136.95 s [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:338]
[OUTPUT] 2025-06-21T15:04:00.511633 Epoch 1 - Block 1 [Training] Accelerator Utilization [AU] (%): 97.9379 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:340]
[OUTPUT] 2025-06-21T15:04:00.511746 Epoch 1 - Block 1 [Training] Throughput (samples/second): 75.0075 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:341]
[OUTPUT] 2025-06-21T15:04:00.511821 Epoch 1 - Block 1 [Training] Computation time per step (second): 0.6366+/-0.0001 (set value: {'mean': 0.636}) [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:342]
2025-06-21 15:16:42|WARNING: Received signal SIGINT (2)
2025-06-21 15:16:42|INFO: Exiting immediately due to SIGTERM
2025-06-21 15:16:42|ERROR: Error occurred while executing command: INTERRUPTED
2025-06-21 15:16:42|STATUS: Writing metadata for benchmark to: /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_150135/training_20250621_150135_metadata.json
2025-06-21 15:16:42|ERROR: Benchmark failed after 1 iterations
Setting attr from num_accelerators to 7
Hosts is: ['192.168.20.21']
Hosts is: ['192.168.20.21']

8x a100

[OUTPUT] 2025-06-21T15:53:27.458607 Saved outputs in /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_154318 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:204]
[OUTPUT] Averaged metric over all steps/epochs
[METRIC] ==========================================================
[METRIC] Number of Simulated Accelerators: 8
[METRIC] Training Accelerator Utilization [AU] (%): 97.6416 (0.1626)
[METRIC] Training Throughput (samples/second): 85.3993 (0.1422)
[METRIC] Training I/O Throughput (MB/second): 11939.6082 (19.8859)
[METRIC] train_au_meet_expectation: success
[METRIC] ==========================================================
 [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:226]
[OUTPUT] 2025-06-21T15:53:27.469267 outputs saved in RANKID_output.json [/root/.venvs/myenv/lib/python3.10/site-packages/dlio_benchmark/utils/statscounter.py:456]
2025-06-21 15:53:46|STATUS: Writing metadata for benchmark to: /local/tmp/mlperf_storage_results/training/unet3d/run/20250621_154318/training_20250621_154318_metadata.json
Setting attr from num_accelerators to 8
Hosts is: ['192.168.20.21']
Hosts is: ['192.168.20.21']

cmd line I'm using is:

mlpstorage training run --model unet3d --data-dir /mnt/mlperf_train/ --oversubscribe --allow-run-as-root --param dataset.num_files_train=10000 --param reader.odirect=True --param reader.read_threads=12 --client-host-memory-in-gb 256 --exec-type mpi --accelerator-type a100 --hosts 192.168.20.21 --num-accelerators 8