when i installed nvshmem_src, nccl get error, how to fix?

W0429 15:49:19.326000 140046767908672 torch/distributed/run.py:779]
W0429 15:49:19.326000 140046767908672 torch/distributed/run.py:779] *****************************************
W0429 15:49:19.326000 140046767908672 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0429 15:49:19.326000 140046767908672 torch/distributed/run.py:779] *****************************************
[rank9]: Traceback (most recent call last):
[rank9]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 30, in
[rank9]: test_all_reduce(rank, local_rank, world_size)
[rank9]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 15, in test_all_reduce
[rank9]: dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank9]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
[rank9]: return func(*args, **kwargs)
[rank9]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2288, in all_reduce
[rank9]: work = group.allreduce([tensor], opts)
[rank9]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:275, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5
[rank9]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
[rank9]: Last error:
[rank9]: [Proxy Service 9] Failed to execute operation Connect from rank 9, retcode 3
[rank10]: Traceback (most recent call last):
[rank10]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 30, in
[rank10]: test_all_reduce(rank, local_rank, world_size)
[rank10]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 15, in test_all_reduce
[rank10]: dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank10]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
[rank10]: return func(*args, **kwargs)
[rank10]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2288, in all_reduce
[rank10]: work = group.allreduce([tensor], opts)
[rank10]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:275, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5
[rank10]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
[rank10]: Last error:
[rank10]: [Proxy Service 10] Failed to execute operation Connect from rank 10, retcode 3
[rank15]: Traceback (most recent call last):
[rank15]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 30, in
[rank15]: test_all_reduce(rank, local_rank, world_size)
[rank15]: File "/sharedata/msm/workspace/DeepEP/test_nccl_rank.py", line 15, in test_all_reduce
[rank15]: dist.all_reduce(data, op=dist.ReduceOp.SUM)
[rank15]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
[rank15]: return func(*args, **kwargs)
[rank15]: File "/home/aigc/miniforge3/envs/mamba/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2288, in all_reduce
[rank15]: work = group.allreduce([tensor], opts)
[rank15]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/NCCLUtils.hpp:275, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5
[rank15]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
[rank15]: Last error:
[rank15]: [Proxy Service 15] Failed to execute operation Connect from rank 15, retcode 3