RuntimeError: Timed out initializing process group in store based barrier on rank 2

I am trying to run pretrain of LLaMA 30b. And here is my running cmd:

deepspeed trainer_sft.py --configs defaults llama-30b-pretrain pretrain --cache_dir $DATA_PATH --output_dir $MODEL_PATH/llama-30b-pre --deepspeed

And after the model was loaded, it stucked for a long time(I think it was 30 mins for the default timeout of pytorch is 30mins).
And this error is raised:

RuntimeError: Timed out initializing process group in store based barrier on rank 2 # for all rank

Any solutions?

Start by reproducing the deepspeed trainer_sft.py command with the listed LLaMA pretraining configurations and inspect the process-group initialization failure on rank 2. Compare the distributed launch and timeout behavior across ranks; done means identifying the initialization cause and completing model startup without the store-based barrier timeout.