Run test_low_latency.py test get hung in H20-3e

Environment details:
- GPU: H20-3e
- Nccl/Cuda version: NCCL version 2.20.5+cuda12.4
- Pytorch version: 2.3.1
- Nivida driver version: 550.127.08, with IBGDA enbale

In my environment, I can run test_internode.py with 4 nodes successfully. But when I run test_low_latency.py, it hangs. The last node output log is as follows:
```
19:54:59.093 node04:4120:4120 [5] NVSHMEM INFO [29] cuIpcOpenMemHandle tobuf 0x1c020000000
19:54:59.093 node04:4121:4121 [6] NVSHMEM INFO [30] cuIpcOpenMemHandle fromhandle 0x32010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.093 node04:4121:4121 [6] NVSHMEM INFO [30] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.093 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle fromhandle 0x29010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.093 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle tobuf 0x1a020000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle fromhandle 0x33010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle tobuf 0x1a020000000
19:54:59.094 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle fromhandle 0x2a010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle tobuf 0x1c020000000
19:54:59.094 node04:4116:4116 [1] NVSHMEM INFO [25] cuIpcOpenMemHandle fromhandle 0x2f010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4116:4116 [1] NVSHMEM INFO [25] cuIpcOpenMemHandle tobuf 0x1c020000000
19:54:59.094 node04:4120:4120 [5] NVSHMEM INFO [29] cuIpcOpenMemHandle fromhandle 0x2d010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4120:4120 [5] NVSHMEM INFO [29] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.094 node04:4115:4115 [0] NVSHMEM INFO [24] cuIpcOpenMemHandle fromhandle 0x33010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4115:4115 [0] NVSHMEM INFO [24] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle fromhandle 0x2d010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle tobuf 0x1c020000000
19:54:59.094 node04:4116:4116 [1] NVSHMEM INFO [25] cuIpcOpenMemHandle fromhandle 0x29010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4116:4116 [1] NVSHMEM INFO [25] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.094 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle fromhandle 0x2b010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4118:4118 [3] NVSHMEM INFO [27] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle fromhandle 0x2e010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
19:54:59.094 node04:4117:4117 [2] NVSHMEM INFO [26] cuIpcOpenMemHandle tobuf 0x1e020000000
19:54:59.481 /deepepdir/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 28 nranks 32 tag 0 - ENTER
19:54:59.481 /deepepdir/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 28 nranks 32 tag 5 - DONE
19:54:59.481 WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.
```

There are some exceptions in the last node NCCL log:

```
2025-07-25 19:54:57.101122 node04:4119:4119 [4] NCCL INFO AllGather: opCount 1 sendbuff 0xb09e00000 recvbuff 0xb09e00800 count 5 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:57.104187 node04:4119:4119 [4] NCCL INFO AllGather: opCount 2 sendbuff 0xb09e00200 recvbuff 0xb09e00600 count 8 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:57.106029 node04:4119:4119 [4] NCCL INFO AllGather: opCount 3 sendbuff 0xb09e00000 recvbuff 0xb09e01400 count 108 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:57.108492 node04:4119:4119 [4] NCCL INFO AllGather: opCount 4 sendbuff 0xb09e00200 recvbuff 0xb09e00600 count 8 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:57.111227 node04:4119:4119 [4] NCCL INFO AllGather: opCount 5 sendbuff 0xb09e00600 recvbuff 0xb09e01e00 count 172 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:59.159245 node04:4119:4119 [4] NCCL INFO AllGather: opCount 6 sendbuff 0xb09e25000 recvbuff 0xb09e70c00 count 8192 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:59.363683 node04:4119:4119 [4] NCCL INFO AllGather: opCount 7 sendbuff 0xb09e25000 recvbuff 0xb09eb0c00 count 8192 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:59.440603 node04:4119:4119 [4] NCCL INFO AllGather: opCount 8 sendbuff 0xb09e25000 recvbuff 0xb09ef0c00 count 8192 datatype 0 op 0 root 0 comm 0x905f350 [nranks=32] stream 0x84ed2f0
2025-07-25 19:54:59.772080 node04:4119:4842 [4] NCCL INFO [Proxy Service UDS] exit: stop 0 abortFlag 1
2025-07-25 19:54:59.832230 node04:4119:4840 [4] NCCL INFO [Service thread] Connection closed by localRank 4
```

And the process stack is as follow:
```
$ pstack 4119
Thread 89 (Thread 0x7ff5dd789700 (LWP 4987) "cuda-EvtHandlr"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff995abc1ef in ?? () from /usr/lib64/libcuda.so.1
#2 0x00007ff995b8064f in ?? () from /usr/lib64/libcuda.so.1
#3 0x00007ff995ab3ee3 in ?? () from /usr/lib64/libcuda.so.1
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 88 (Thread 0x7ff63e823700 (LWP 4986) "pt_main_thread"):
#0 0x00007ffa2a821017 in pthread_join () from /usr/lib64/libpthread.so.0
#1 0x00007ff9c5a48dc7 in commFree (comm=0x84dc290) at init.cc:183
#2 commCleanup (comm=0x84dc290) at init.cc:1950
#3 commReclaim (comm=comm@entry=0x84dc290) at init.cc:2081
#4 0x00007ff9c5a4c867 in ncclCommAbort (comm=0x84dc290) at init.cc:2149
#5 0x00007ff9efcebed2 in c10d::NCCLComm::ncclCommAbort(std::optional) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#6 0x00007ff9efcbb2db in c10d::ProcessGroupNCCL::abortCommsFromMap(std::unordered_map, std::hash, std::equal_to, std::allocator > > >&, std::optional) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#7 0x00007ff9efcbb62c in c10d::ProcessGroupNCCL::abort(std::optional) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#8 0x00007ff9efcbcf0e in std::_Function_handler (), std::__future_base::_Task_setter, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker)::{lambda()#1}> >, bool> >::_M_invoke(std::_Any_data const&) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#9 0x00007ff9efb8125b in std::__future_base::_State_baseV2::_M_do_set(std::function ()>*, bool*) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#10 0x00007ffa2a81e20b in __pthread_once_slow () from /usr/lib64/libpthread.so.0
#11 0x00007ff9efcb2af4 in std::__future_base::_Async_state_impl)::{lambda()#1}> >, bool>::_M_run() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#12 0x00007ffa297fbbf4 in std::execute_native_thread_routine (__p=0x5421830) at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82
#13 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#14 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 87 (Thread 0x7ff67cef7700 (LWP 4981) "pt_main_thread"):
#0 0x00007ff8ed3e274c in nvshmemi_proxy_progress_minimal(void*) () from /deepepdir/nvshmem_src/installed/lib/libnvshmem_host.so.3
#1 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#2 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 86 (Thread 0x7ff670ffd700 (LWP 4947) "pt_main_thread"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff9c5a59310 in ncclProxyGetPostedOps (added=, proxyState=0x7ff650011810) at proxy.cc:728
#2 ncclProxyProgress (proxyState_=0x7ff650011810) at proxy.cc:884
#3 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 85 (Thread 0x7ff6717fe700 (LWP 4927) "pt_main_thread"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff9c5a5812a in ncclProxyServiceUDS (_args=0x7ff650011810) at proxy.cc:1634
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 84 (Thread 0x7ff671fff700 (LWP 4925) "pt_main_thread"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff9c5a5997f in ncclProxyService (_args=0x7ff650011810) at proxy.cc:1485
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 83 (Thread 0x7ff67d6f8700 (LWP 4895) "pt_main_thread"):
#0 0x00007ffa2a823de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff9efcb680d in c10d::ProcessGroupNCCL::heartbeatMonitor() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#2 0x00007ffa297fbbf4 in std::execute_native_thread_routine (__p=0x7ff688bf10b0) at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82
#3 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 82 (Thread 0x7ff6c8f37700 (LWP 4894) "pt_main_thread"):
#0 0x00007ffa2a823de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff9efcbb906 in c10d::ProcessGroupNCCL::watchdogHandler() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#2 0x00007ff9efcbc9ff in c10d::ProcessGroupNCCL::ncclCommWatchdog() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#3 0x00007ffa297fbbf4 in std::execute_native_thread_routine (__p=0x86e80e0) at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 81 (Thread 0x7ff67ffff700 (LWP 4855) "pt_main_thread"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff9c5a59310 in ncclProxyGetPostedOps (added=, proxyState=0x7ff688d51490) at proxy.cc:728
#2 ncclProxyProgress (proxyState_=0x7ff688d51490) at proxy.cc:884
#3 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 80 (Thread 0x7ff685652700 (LWP 4840) "pt_main_thread"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff9c5a5997f in ncclProxyService (_args=0x7ff688d51490) at proxy.cc:1485
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 79 (Thread 0x7ff686ffd700 (LWP 4754) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff686ffce70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc680 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 78 (Thread 0x7ff6877fe700 (LWP 4750) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6877fde70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc580 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 77 (Thread 0x7ff687fff700 (LWP 4746) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff687ffee70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc480 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 76 (Thread 0x7ff68cffd700 (LWP 4741) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff68cffce70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc380 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 75 (Thread 0x7ff68d7fe700 (LWP 4737) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff68d7fde70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc280 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 74 (Thread 0x7ff68dfff700 (LWP 4733) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff68dffee70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc180 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 73 (Thread 0x7ff6928db700 (LWP 4729) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6928dae70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fc080 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 72 (Thread 0x7ff6930dc700 (LWP 4725) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6930dbe70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fbf80 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 71 (Thread 0x7ff6938dd700 (LWP 4721) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6938dce70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fbe80 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 70 (Thread 0x7ff6a0ffd700 (LWP 4717) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6a0ffce70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fbd80 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 69 (Thread 0x7ff6a17fe700 (LWP 4713) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6a17fde70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fbc80 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 68 (Thread 0x7ff6a1fff700 (LWP 4710) "pt_main_thread"):
#0 0x00007ffa2a82675d in read () from /usr/lib64/libpthread.so.0
#1 0x00007ff6e038e81f in ibv_get_async_event () from /usr/lib64/libibverbs.so
#2 0x00007ff9c5a89232 in wrap_ibv_get_async_event (context=, event=event@entry=0x7ff6a1ffee70) at misc/ibvwrap.cc:121
#3 0x00007ff9c5a9df6c in ncclIbAsyncThreadMain (args=0x7ff9cb4fbb80 ) at transport/net_ib.cc:96
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 67 (Thread 0x7ff6ca759700 (LWP 4678) "cuda-EvtHandlr"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff995abc1ef in ?? () from /usr/lib64/libcuda.so.1
#2 0x00007ff995b8064f in ?? () from /usr/lib64/libcuda.so.1
#3 0x00007ff995ab3ee3 in ?? () from /usr/lib64/libcuda.so.1
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 66 (Thread 0x7ff6caf5a700 (LWP 4677) "pt_main_thread"):
#0 0x00007ffa2a823de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff9efcb680d in c10d::ProcessGroupNCCL::heartbeatMonitor() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#2 0x00007ffa297fbbf4 in std::execute_native_thread_routine (__p=0x7ff6c40008c0) at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/thread.cc:82
#3 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#4 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 65 (Thread 0x7ff8ecc87700 (LWP 4636) "cuda0000240000a"):
#0 0x00007ffa29e34ddd in poll () from /usr/lib64/libc.so.6
#1 0x00007ff995abc1ef in ?? () from /usr/lib64/libcuda.so.1
#2 0x00007ff995b8064f in ?? () from /usr/lib64/libcuda.so.1
#3 0x00007ff995ab3ee3 in ?? () from /usr/lib64/libcuda.so.1
#4 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#5 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 64 (Thread 0x7ff9154b7700 (LWP 4311) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 63 (Thread 0x7ff915cb8700 (LWP 4310) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 62 (Thread 0x7ff9164b9700 (LWP 4309) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 61 (Thread 0x7ff916cba700 (LWP 4308) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 60 (Thread 0x7ff9174bb700 (LWP 4307) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 59 (Thread 0x7ff917cbc700 (LWP 4306) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 58 (Thread 0x7ff9184bd700 (LWP 4305) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 57 (Thread 0x7ff91acbe700 (LWP 4304) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 56 (Thread 0x7ff91d4bf700 (LWP 4303) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 55 (Thread 0x7ff91fcc0700 (LWP 4302) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 54 (Thread 0x7ff9224c1700 (LWP 4301) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 53 (Thread 0x7ff924cc2700 (LWP 4299) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 52 (Thread 0x7ff9274c3700 (LWP 4297) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 51 (Thread 0x7ff929cc4700 (LWP 4295) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 50 (Thread 0x7ff92e4c5700 (LWP 4293) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 49 (Thread 0x7ff930cc6700 (LWP 4290) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 48 (Thread 0x7ff9334c7700 (LWP 4288) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 47 (Thread 0x7ff933cc8700 (LWP 4286) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 46 (Thread 0x7ff9364c9700 (LWP 4285) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 45 (Thread 0x7ff938cca700 (LWP 4282) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 44 (Thread 0x7ff93b4cb700 (LWP 4280) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 43 (Thread 0x7ff93dccc700 (LWP 4278) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 42 (Thread 0x7ff9404cd700 (LWP 4276) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 41 (Thread 0x7ff942cce700 (LWP 4274) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 40 (Thread 0x7ff9454cf700 (LWP 4272) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 39 (Thread 0x7ff947cd0700 (LWP 4270) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 38 (Thread 0x7ff94a4d1700 (LWP 4268) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 37 (Thread 0x7ff94ccd2700 (LWP 4266) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 36 (Thread 0x7ff94f4d3700 (LWP 4264) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 35 (Thread 0x7ff951cd4700 (LWP 4262) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 34 (Thread 0x7ff9544d5700 (LWP 4259) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 33 (Thread 0x7ff956cd6700 (LWP 4257) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 32 (Thread 0x7ff9594d7700 (LWP 4255) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 31 (Thread 0x7ff95bcd8700 (LWP 4253) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 30 (Thread 0x7ff95e4d9700 (LWP 4250) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 29 (Thread 0x7ff960cda700 (LWP 4248) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 28 (Thread 0x7ff9634db700 (LWP 4245) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 27 (Thread 0x7ff965cdc700 (LWP 4243) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 26 (Thread 0x7ff9684dd700 (LWP 4241) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 25 (Thread 0x7ff968cde700 (LWP 4238) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 24 (Thread 0x7ff96b4df700 (LWP 4237) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 23 (Thread 0x7ff96bce0700 (LWP 4235) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 22 (Thread 0x7ff96c4e1700 (LWP 4232) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 21 (Thread 0x7ff96ece2700 (LWP 4230) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 20 (Thread 0x7ff9714e3700 (LWP 4228) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 19 (Thread 0x7ff971ce4700 (LWP 4226) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 18 (Thread 0x7ff9744e5700 (LWP 4224) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 17 (Thread 0x7ff976ce6700 (LWP 4222) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 16 (Thread 0x7ff9774e7700 (LWP 4220) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 15 (Thread 0x7ff979ce8700 (LWP 4218) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 14 (Thread 0x7ff97c4e9700 (LWP 4215) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 13 (Thread 0x7ff97ccea700 (LWP 4213) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 12 (Thread 0x7ff97f4eb700 (LWP 4211) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 11 (Thread 0x7ff981cec700 (LWP 4209) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 10 (Thread 0x7ff9844ed700 (LWP 4207) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 9 (Thread 0x7ff984cee700 (LWP 4205) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 8 (Thread 0x7ff9854ef700 (LWP 4203) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 7 (Thread 0x7ff987cf0700 (LWP 4200) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 6 (Thread 0x7ff98a4f1700 (LWP 4198) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 5 (Thread 0x7ff98ccf2700 (LWP 4196) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 4 (Thread 0x7ff98f4f3700 (LWP 4194) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 3 (Thread 0x7ff991cf4700 (LWP 4191) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 2 (Thread 0x7ff9924f5700 (LWP 4189) "python3"):
#0 0x00007ffa2a823a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
#1 0x00007ff992f40f8b in blas_thread_server () from /usr/local/conda/lib/python3.9/site-packages/numpy/_core/../../numpy.libs/libscipy_openblas64_-99b71e71.so
#2 0x00007ffa2a81fea5 in start_thread () from /usr/lib64/libpthread.so.0
#3 0x00007ffa29e3fb0d in clone () from /usr/lib64/libc.so.6
Thread 1 (Thread 0x7ffa2b06f7c0 (LWP 4119) "pt_main_thread"):
#0 0x00007ffa29e39e29 in syscall () from /usr/lib64/libc.so.6
#1 0x00007ffa297f97e2 in std::__atomic_futex_unsigned_base::_M_futex_wait_until_steady (this=, __addr=0x6000a50, __val=2147483648, __has_timeout=, __s=..., __ns=...) at /opt/conda/conda-bld/gcc-compiler_1654084175708/work/gcc/libstdc++-v3/src/c++11/futex.cc:217
#2 0x00007ff9efcb4b41 in c10d::ProcessGroupNCCL::waitForFutureOrTimeout(std::future&, std::chrono::duration > const&, std::string const&, bool) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#3 0x00007ff9efcb79fa in c10d::ProcessGroupNCCL::shutdown(std::optional) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#4 0x00007ff9efcb8390 in c10d::ProcessGroupNCCL::~ProcessGroupNCCL() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#5 0x00007ff9efcb8b49 in c10d::ProcessGroupNCCL::~ProcessGroupNCCL() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so
#6 0x00007ffa19e635da in c10d::ProcessGroup::release_resources() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#7 0x00007ffa284eb938 in c10::intrusive_ptr >::reset_() () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#8 0x00007ffa284eb99c in pybind11::class_ >, c10d::PyProcessGroup>::dealloc(pybind11::detail::value_and_holder&) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#9 0x00007ffa27ccda76 in pybind11::detail::clear_instance(_object*) () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#10 0x00007ffa27cce931 in pybind11_object_dealloc () from /usr/local/conda/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#11 0x00000000004e4166 in _Py_Dealloc (op=) at /usr/local/src/conda/python-3.9.20/Objects/object.c:2209
#12 _Py_DECREF (op=) at /usr/local/src/conda/python-3.9.20/Include/object.h:430
#13 _Py_XDECREF (op=) at /usr/local/src/conda/python-3.9.20/Include/object.h:497
#14 dict_dealloc (mp=0x7ff914c00140) at /usr/local/src/conda/python-3.9.20/Objects/dictobject.c:2018
#15 0x00000000005091e4 in _Py_Dealloc (op=) at /usr/local/src/conda/python-3.9.20/Objects/object.c:2209
#16 _Py_DECREF (op=) at /usr/local/src/conda/python-3.9.20/Include/object.h:430
#17 subtype_dealloc (self=0x7ff914bfb2e0) at /usr/local/src/conda/python-3.9.20/Objects/typeobject.c:1331
#18 0x0000000000592808 in _Py_Dealloc (op=) at /usr/local/src/conda/python-3.9.20/Objects/object.c:2209
#19 _Py_DECREF (op=) at /usr/local/src/conda/python-3.9.20/Include/object.h:430
#20 _Py_XDECREF (op=) at /usr/local/src/conda/python-3.9.20/Include/object.h:497
#21 free_keys_object (keys=0x6120900) at /usr/local/src/conda/python-3.9.20/Objects/dictobject.c:598
#22 0x00000000005b5776 in dict_tp_clear (op=) at /usr/local/src/conda/python-3.9.20/Objects/dictobject.c:3158
#23 0x00000000004dd270 in delete_garbage (old=, collectable=0x7ffd7380f0a0, gcstate=0x21c1bc8, tstate=0x21c3400) at /usr/local/src/conda/python-3.9.20/Modules/gcmodule.c:1004
#24 collect (tstate=0x21c3400, generation=2, n_collected=0x0, n_uncollectable=0x0, nofail=1) at /usr/local/src/conda/python-3.9.20/Modules/gcmodule.c:1273
#25 0x00000000005c2da7 in _PyGC_CollectNoFail () at /usr/local/src/conda/python-3.9.20/Modules/gcmodule.c:2109
#26 0x00000000005c1559 in Py_FinalizeEx () at /usr/local/src/conda/python-3.9.20/Python/pylifecycle.c:1426
#27 0x00000000005c1a68 in Py_Exit (sts=1) at /usr/local/src/conda/python-3.9.20/Python/pylifecycle.c:2433
#28 0x00000000005befdb in handle_system_exit () at /usr/local/src/conda/python-3.9.20/Python/pythonrun.c:723
#29 0x00000000005bed52 in _PyErr_PrintEx (set_sys_last_vars=1, tstate=0x21c3400) at /usr/local/src/conda/python-3.9.20/Python/pythonrun.c:733
#30 PyErr_PrintEx (set_sys_last_vars=1) at /usr/local/src/conda/python-3.9.20/Python/pythonrun.c:828
#31 0x00000000005bed37 in PyErr_Print () at /usr/local/src/conda/python-3.9.20/Python/pythonrun.c:834
#32 0x00000000005b4fb5 in PyRun_SimpleStringFlags (command=0x7ffa2aef8e50 "from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=6, pipe_handle=16)\n", flags=0x7ffd7380f228) at /usr/local/src/conda/python-3.9.20/Python/pythonrun.c:499
#33 0x00000000005b47ec in pymain_run_command (cf=0x7ffd7380f228, command=) at /usr/local/src/conda/python-3.9.20/Modules/main.c:252
#34 pymain_run_python (exitcode=0x7ffd7380f220) at /usr/local/src/conda/python-3.9.20/Modules/main.c:599
#35 Py_RunMain () at /usr/local/src/conda/python-3.9.20/Modules/main.c:687
#36 0x0000000000587fd9 in Py_BytesMain (argc=, argv=) at /usr/local/src/conda/python-3.9.20/Modules/main.c:1133
#37 0x00007ffa29d63555 in __libc_start_main () from /usr/lib64/libc.so.6
#38 0x0000000000587e8e in _start () at /usr/local/src/conda/python-3.9.20/Python/getplatform.c:9
```

Any idea about this problem? Thanks.