[BUG] Code blocking when training on multi-nodes using DS-Chat.

Describe the bug
When using multi-nodes to train DS-Chat, it will happen the code blocking util time-out.
To Reproduce

1. I use SSH key to realize passwordless SSH connection:
   ssh-keygen xxxx to generate the private and public key in ~/.ssh/
ssh-copy-id -i ~/.ssh/id_rsa.pub xxx to send the public key to other nodes
2. I configure the alias of each node in ~/.ssh/config

Host worker-i
    Hostname xxxx
    user xxxx

3. write the all node in hostfile:

worker-i slots=xxx

4. using deepspeed to launch the distributed training using DS-Chat to realize the stage-1 SFT:

deepspeed --hostfile=hostfile ./applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py xxx

ds_report output
By default, deepspeed will select the first node in hostfile as master node, so I execute the forth operation in the first node. After repeatedly executing the forth operation and Ctrl + C to kill it. When executed the forth operation again later, it will be blocking util time-out, and report the error as follows:

torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout

Were previously killed processes not killed completely, piling up together? Because I see the previous processes using top:

Reproduce the multi-node launch from applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py using the SSH configuration and hostfile described in the report. Start by checking the processes left after Ctrl+C and the distributed launcher output, then determine what causes the NCCL store timeout on a later launch. Done means the repeated launch no longer blocks until Socket Timeout.