TCPStore connection timeout during GRPO training.

Rollout bash:

#!/bin/bash
export WANDB_MODE=offline
ROLLOUT_GPUS=(0 1)
RLHF_GPUS=(2 3 4 5 6 7)
MEM_THRESHOLD=1500 # MB

WANDB_API_KEY="xxxxxxx"
MODEL_PATH="xxx"
DATA_PATH="xxx"
export CUDA_VISIBLE_DEVICES=$(IFS=,; echo "${ROLLOUT_GPUS[*]}");
swift rollout
--model $MODEL_PATH
--multi_turn_scheduler thinkscheduler
--model_type deepseek_r1_distill
--tensor_parallel_size 2
--data_parallel_size 1
--max_turns 5
--external_plugins examples/train/grpo/plugin/multi.py \

Rlhf bash:

export WANDB_MODE=offline
ROLLOUT_GPUS=(0 1)
RLHF_GPUS=(2 3 4 5 6 7)
MEM_THRESHOLD=1500 # MB

WANDB_API_KEY="xxx"
MODEL_PATH="xxx"
DATA_PATH="xxxx"
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY
export WANDB_API_KEY="$WANDB_API_KEY";
export CUDA_VISIBLE_DEVICES=$(IFS=,; echo "${RLHF_GPUS[*]}");
export NPROC_PER_NODE=${#RLHF_GPUS[@]};

swift rlhf
--rlhf_type grpo
--model $MODEL_PATH
--external_plugins examples/train/grpo/plugin/multi.py
--reward_funcs xxxx
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8602
--vllm_server_timeout 10
--train_type lora
--torch_dtype bfloat16
--dataset $DATA_PATH
--model_type deepseek_r1_distill
--num_generations 4
--max_turns 5
--num_train_epochs 3
--per_device_train_batch_size 2
--per_device_eval_batch_size 4
--gradient_accumulation_steps 8
--use_vllm true
--max_length 2048
--max_completion_length 2048
--learning_rate 1e-5
--save_strategy epoch
--eval_strategy epoch
--save_total_limit 10
--logging_steps 1
--warmup_ratio 0.05
--dataloader_num_workers 4
--temperature 0.5
--deepspeed zero3
--log_completions true
--report_to wandb
--beta 0.001
--num_iterations 1
--output_dir xxxx
--multi_turn_scheduler thinkscheduler
--max_turn 5

Error：
[E1002 10:48:53.023166267 socket.cpp:1019] [c10d] The client socket has timed out after 300000ms while trying to connect to (127.0.0.1, 51216). [E1002 10:48:53.023389940 TCPStore.cpp:331] [c10d] TCP client failed to connect/validate to host 127.0.0.1:51216 - timed out (try=1, timeout=300000ms): The client socket has timed out after 300000ms while trying to connect to (127.0.0.1, 51216). Exception raised from throwTimeoutError at /pytorch/torch/csrc/distributed/c10d/socket.cpp:1021 (most recent call first)

Conda environment: I have tried both torch 2.7.0 + vllm 0.9.2 + deepspeed 0.15.4 and torch 2.8.0 + vllm 0.10.1.1 + deepspeed 0.17.2, but neither worked. However, swift 3.6.0 can run GRPO training normally, while swift 3.8.3 and 3.9.0 produce the above error.

I also found that other developers are experiencing the same issue as me:
>

Error trace location:
ms-swift/swift/trainers/rlhf_trainer/vllm_client.py

> pg = StatelessProcessGroup.create(
>                 host=self.hosts[i],
>                 port=self.group_ports[i],
>                 rank=rank,
>                 world_size=world_size,
>             )

python3.11/site-packages/vllm/distributed/utils.py

> store = TCPStore(
>     host_name=host,
>     port=port,
>     world_size=world_size,
>     is_master=launch_server,
>     timeout=timedelta(seconds=store_timeout),
>     use_libuv=False,  # for now: github.com/pytorch/pytorch/pull/150215
>     master_listen_fd=listen_fd,
> )

Start with swift/trainers/rlhf_trainer/vllm_client.py at the StatelessProcessGroup.create call, then compare it with vllm's distributed/utils.py TCPStore setup. Reproduce the reported GRPO commands using the listed swift, torch, vLLM, and DeepSpeed versions, including the working swift 3.6.0 comparison. Done means swift 3.8.3 or 3.9.0 completes GRPO training without the TCPStore connection timeout.