[BUG] Failed for using cpu for pipeline based training across multiple machines (2 machines actually)

Describe the bug

I have two ubuntu machines, and with 10Gb/s erthnet cable connected and I want to use deepspeed to use these two machines to
run a model training with pipeline parallel, and only with cpu to do the training.

Both of these machines installed pytorch version 1.13.1. And:
deepspeed cpu accelerator related dependencies are installed like this:

1. pip install intel-extension-for-pytorch==1.13.100
2. python -m pip install oneccl_bind_pt==1.13 -f https://developer.intel.com/ipex-whl-stable-cpu
3. git clone https://github.com/oneapi-src/oneCCL # And do cmake based compiling and installation.
   And the deepspeed version is: 0.14.0

And The command line for launching the training:

deepspeed --master_addr=192.168.23.110  --num_nodes=2  --hostfile=./hostfile_linux  --master_port=29555  pipeline_model.py

And the content of ./hostfile_linux:

worker-alienbook-wl slots=1
worker-3090ti-wl slots=1

After changing various arguments, building soft links by 'ln -s' to create the same accessing path for files and directory on both machines， And even change some installed library code (like torch and deepspeed). I thought it should work finally. But I could I'm still stuck by one line in my own pipeline_model.py

deepspeed.init_distributed(dist_backend="ccl")

Both of my machines could enter this function, but they never return from this function .

I tried my best to investigate the issue, but no break through. Any help is appreciated. And maybe I should report this issue in pytorch intel extension project. But it looks like there is not much people there :(. After stucking more than 120 seconds, some backtraces are displayed.

Here is the related log messages arround:

----- omit some unrelated messages -----
worker-alienbook-wl: Using /home/axu/.cache/torch_extensions/py39_cu116 as PyTorch extensions root...
worker-alienbook-wl: Emitting ninja build file /home/axu/.cache/torch_extensions/py39_cu116/deepspeed_ccl_comm/build.ninja...
worker-alienbook-wl: Building extension module deepspeed_ccl_comm...
worker-alienbook-wl: Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
worker-alienbook-wl: ninja: no work to do.
worker-alienbook-wl: Loading extension module deepspeed_ccl_comm...
worker-alienbook-wl: Time to load deepspeed_ccl_comm op: 0.09088420867919922 seconds
worker-alienbook-wl: DeepSpeed deepspeed.ops.comm.deepspeed_ccl_comm_op built successfully
worker-alienbook-wl: 2024-03-26 18:28:02,528 - torch.distributed.distributed_c10d - INFO - Added key: store_based_barrier_key:1 to store for rank: 0
worker-alienbook-wl: 2024-03-26 18:28:02,530 - torch.distributed.distributed_c10d - INFO - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
worker-alienbook-wl: 2024:03:26-18:28:02:(43479) |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
worker-3090ti-wl: 2024-03-26 18:28:02,591 - torch.distributed.distributed_c10d - INFO - Added key: store_based_barrier_key:1 to store for rank: 1
worker-3090ti-wl: 2024-03-26 18:28:02,598 - torch.distributed.distributed_c10d - INFO - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
worker-3090ti-wl: 2024:03:26-18:28:02:(75330) |CCL_WARN| did not find MPI-launcher specific variables, switch to ATL/OFI, to force enable ATL/MPI set CCL_ATL_TRANSPORT=mpi
worker-3090ti-wl: write: error: buf 0x558f0a2e4310, size 394, shift 0
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| internal_kvs.cpp:529 kvs_init: connection time (131) >= limit (120)
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| internal_kvs_server.hpp:66 put: read/write error: Broken pipe
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| internal_kvs.cpp:108 kvs_get_value_by_name_key: client: get_value
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| pmi_resizable_simple_internal.cpp:319 get_local_kvs_id: failed to get local kvs id
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| pmi_resizable_simple_internal.cpp:65 pmrt_init: failed to get local id
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| atl_ofi_comm.cpp:274 init_transport: pmi init failed
worker-3090ti-wl: 2024:03:26-18:30:13:(75330) |CCL_ERROR| atl_ofi_comm.cpp:79 atl_ofi_comm: condition init_transport(true) == ATL_STATUS_SUCCESS failed
worker-3090ti-wl: init transport failed
worker-3090ti-wl: Traceback (most recent call last):
worker-3090ti-wl:   File "/home/ubuntu/proj/minGPT/pipeline_model.py", line 367, in <module>
worker-3090ti-wl:     main()
worker-3090ti-wl:   File "/home/ubuntu/proj/minGPT/pipeline_model.py", line 309, in main
worker-3090ti-wl:     deepspeed.init_distributed(dist_backend="ccl")
worker-3090ti-wl:   File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 635, in init_distributed
worker-3090ti-wl:     init_deepspeed_backend(get_accelerator().communication_backend_name(), timeout, init_method)
worker-3090ti-wl:   File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 160, in init_deepspeed_backend
worker-3090ti-wl:     ccl_backend = CCLBackend(rank=rank, world_size=size, timeout=timeout, init_method=init_method)
worker-3090ti-wl:   File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/deepspeed/comm/ccl.py", line 53, in __init__
worker-3090ti-wl:     super(CCLBackend, self).broadcast(main_kvs, 0)
worker-3090ti-wl:   File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 205, in broadcast
worker-3090ti-wl:     return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
worker-3090ti-wl:   File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1400, in broadcast
worker-3090ti-wl:     work = default_pg.broadcast([tensor], opts)
worker-3090ti-wl: RuntimeError: oneCCL: atl_ofi_comm.cpp:79 atl_ofi_comm: EXCEPTION: init transport failed
worker-3090ti-wl: [2024-03-26 18:30:13,683] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 75330
worker-3090ti-wl: [2024-03-26 18:30:13,684] [ERROR] [launch.py:322:sigkill_handler] ['/home/axu/anaconda3/bin/python', '-u', 'pipeline_model.py', '--local_rank=0'] exits with return code = 1
pdsh@axu-Alienware-15-R4: worker-3090ti-wl: ssh exited with exit code 1

To Reproduce
Steps to reproduce the behavior:
See previous section.

Expected behavior
Pipeline parallel based training thould start normally

ds_report output
Please run ds_report to give us details about your setup.

ds_report result for both machines(the INFO displayed is caused by modification of the deepspeed library code):

[2024-03-26 21:12:52,143] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cpu (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
deepspeed_not_implemented  [NO] ....... [OKAY]
deepspeed_ccl_comm ..... [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/axu/anaconda3/lib/python3.9/site-packages/torch']
torch version .................... 1.13.1
deepspeed install path ........... ['/home/axu/anaconda3/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.14.0, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.13 
shared memory (/dev/shm) size .... 15.60 GB

[2024-03-26 21:28:52,933] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cpu (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
deepspeed_not_implemented  [NO] ....... [OKAY]
deepspeed_ccl_comm ..... [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/ubuntu/anaconda3/lib/python3.9/site-packages/torch']
torch version .................... 1.13.1
deepspeed install path ........... ['/home/ubuntu/anaconda3/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.14.0, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.13 
shared memory (/dev/shm) size .... 62.78 GB

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS: [Ubuntu 20.04 for both machine]
• GPU count and types [one machine with 1080ti and another with two 3090ti, but I did not use any gpus at all for training]
• Interconnects (if applicable) [two machines connected with 10 Gbps ethernet]
• one machine with Python 3.9.7 and another with 3.9.12
• Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

No, Only pdsh laucher are used.

Docker context
Are you using a specific docker image that you can share?

No docker, just on physical machines

Additional context
Any other information you need, I will provide them.

Start with pipeline_model.py at the deepspeed.init_distributed(dist_backend="ccl") call, then read the CCLBackend path in deepspeed/comm/ccl.py and the accompanying traceback. Reproduce with the two-machine hostfile and compare the launcher, oneCCL transport, and environment details shown in the logs. Done means CPU pipeline-parallel initialization completes across both machines instead of timing out in oneCCL.