Timeout message is printed in barrier_block

When we train MoE LLM by going with Megatron empowered by DeepEP for EP communication, Megatron will hang randomly with below error messages printed in the output log:

> DeepEP timeout check failed: rank = 6, thread = 0, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 1, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 2, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 3, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 4, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 5, value = 1024)
> DeepEP timeout check failed: rank = 6, thread = 6, value = 0)
> DeepEP timeout check failed: rank = 6, thread = 7, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 0, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 1, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 2, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 3, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 4, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 5, value = 1024)
> DeepEP timeout check failed: rank = 2, thread = 6, value = 0)
> DeepEP timeout check failed: rank = 2, thread = 7, value = 0)

In this case, with CP enabled, the EP communiction group is unfolded along CP dimention.

By checking DeepEP code, we understood that this message is printed by barrier_block which used to sync up NVL ranks within a node. As shown in the above message, it looks the rank with local id 5 failed to go into the barrier_block.

We can duplicate this situation, within a 2-nodes environment by training a customized small LLM, when get a RNIC port down. But in our large-scaled cluster, we didn't observe the alarms of port up/down events when this problem happened.

In our cluster environment, NVSHMEM 3.3.9 is used.

It would be highly appreciated if any other clue or thought. Many thanks.