[BUG]Single node single GPU work well but single node multi gpu hangs on

Describe the bug
I use this command, deepspeed --include localhost:"1" finetune_wav2vec2_for_ASR.py, to finetune a Wav2Vec2 model using transformers trainer, it works well, but when change --include to --include localhost:"1,3", everything is fine before the training step, when reach it, the program just hangs on, like figure below:

To Reproduce
just deepspeed --include localhost:"1,3" finetune_wav2vec2_for_ASR.py

ds_report output
Please run ds_report to give us details about your setup.

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.8/dist-packages/torch']
torch version .................... 1.12.0+cu102
deepspeed install path ........... ['/usr/local/lib/python3.8/dist-packages/deepspeed']
deepspeed info ................... 0.9.1, unknown, unknown
torch cuda version ............... 10.2
torch hip version ................ None
nvcc version ..................... 10.2
deepspeed wheel compiled w. ...... torch 1.12, cuda 10.2

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS:Ubuntu 18.04
• GPU count and types: one machine with four RTX2080Ti
• Python version:3.8
• Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
deepspeed --include localhost:"1,3" finetune_wav2vec2_for_ASR.py

Start by reproducing the provided finetune_wav2vec2_for_ASR.py command with one GPU and with GPUs 1 and 3, while reviewing the ds_report environment details and the point where training begins. Compare launcher and training-step behavior; the issue is resolved when the multi-GPU run completes training without hanging.