error: nvshmem API called before nvshmem_init

error when run test_low_latency.py:
Allocating buffer size: 2115.111296 MB ...
/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

/root/nvshmem_src/src/host/init/init.cu:nvshmemi_check_state_and_init:1077: nvshmem API called before nvshmem_init

W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887117 via signal SIGTERM
W0425 00:53:56.038000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887118 via signal SIGTERM
W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887120 via signal SIGTERM
W0425 00:53:56.039000 887048 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 887121 via signal SIGTERM
Traceback (most recent call last):
File "/root/DeepEP-main/tests/test_low_latency.py", line 172, in
torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 5 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
fn(i, *args)
File "/root/DeepEP-main/tests/test_low_latency.py", line 156, in test_loop
buffer = deep_ep.Buffer(group, num_rdma_bytes=num_rdma_bytes, low_latency_mode=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/deep_ep-1.0.0+8a0ca8e-py3.11-linux-x86_64.egg/deep_ep/buffer.py", line 59, in __init__
dist.all_gather_object(device_ids, local_device_id, group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3035, in all_gather_object
all_gather(object_size_list, local_size, group=group)
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ds/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3704, in all_gather
work = group.allgather([tensor_list], [tensor])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/NCCLUtils.hpp:268, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.21.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 1 'invalid argument'