deepseek-ai / deepseek-ai/DeepEP
Internode low latency test dispatch & combine finished with error
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
When I ran internode low latency test, the dispatch and combine finished correctly, but the test case dump error printing in the end.
Node 1:
```
root@1e604ac9d157:/home/dpsk_a2a/DeepEP# NCCL_DEBUG=WARN MASTER_ADDR= WORLD_SIZE=2 RANK=0 python3 tests/test_low_latency.py
Allocating buffer size: 2115.111296 MB ...
......
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 0 nranks 16 tag 4 - DONE
[rank 0] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.58 us, min_t=962.43 us, max_t=994.94 us
[rank 1] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.81 us, min_t=961.82 us, max_t=995.49 us
[rank 6] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.93 us, min_t=961.12 us, max_t=997.79 us
[rank 2] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.71 us, min_t=964.48 us, max_t=987.07 us
[rank 4] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.75 us, min_t=960.96 us, max_t=996.83 us
[rank 7] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.71 us, min_t=964.35 us, max_t=993.25 us
[rank 3] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.98 us, min_t=956.42 us, max_t=992.29 us
[rank 5] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.53 us, min_t=954.53 us, max_t=996.86 us
[rank 1] Dispatch bandwidth: 21.22 GB/s, avg_t=353.99 us | Combine bandwidth: 22.93 GB/s, avg_t=634.05 us
[rank 4] Dispatch bandwidth: 21.76 GB/s, avg_t=345.21 us | Combine bandwidth: 22.92 GB/s, avg_t=634.31 us
[rank 0] Dispatch bandwidth: 22.30 GB/s, avg_t=336.84 us | Combine bandwidth: 23.01 GB/s, avg_t=631.76 us
[rank 2] Dispatch bandwidth: 21.51 GB/s, avg_t=349.20 us | Combine bandwidth: 23.01 GB/s, avg_t=631.73 us
[rank 3] Dispatch bandwidth: 21.41 GB/s, avg_t=350.90 us | Combine bandwidth: 22.91 GB/s, avg_t=634.48 us
[rank 6] Dispatch bandwidth: 22.45 GB/s, avg_t=334.54 us | Combine bandwidth: 22.94 GB/s, avg_t=633.66 us
[rank 5] Dispatch bandwidth: 22.08 GB/s, avg_t=340.25 us | Combine bandwidth: 22.92 GB/s, avg_t=634.31 us
[rank 7] Dispatch bandwidth: 22.18 GB/s, avg_t=338.64 us | Combine bandwidth: 23.41 GB/s, avg_t=621.07 us
[rank 1] Dispatch send/recv time: 28.29 us | Combine send/recv time: 28.58 us
[rank 4] Dispatch send/recv time: 28.25 us | Combine send/recv time: 28.81 us
[rank 2] Dispatch send/recv time: 28.45 us | Combine send/recv time: 28.79 us
[rank 3] Dispatch send/recv time: 28.26 us | Combine send/recv time: 28.85 us
[rank 0] Dispatch send/recv time: 27.99 us | Combine send/recv time: 28.84 us
[rank 6] Dispatch send/recv time: 27.97 us | Combine send/recv time: 28.56 us
[rank 5] Dispatch send/recv time: 28.12 us | Combine send/recv time: 28.94 us
[rank 7] Dispatch send/recv time: 28.24 us | Combine send/recv time: 29.03 us
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 4 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 0 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 2 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 5 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 7 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 1 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 6 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 3 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 6 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 7 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 5 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 3 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 1 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 2 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 4 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 0 nranks 16 tag 4 - DONE
W0513 15:52:40.025000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7230 via signal SIGTERM
W0513 15:52:40.026000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7231 via signal SIGTERM
W0513 15:52:40.026000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7233 via signal SIGTERM
W0513 15:52:40.027000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7234 via signal SIGTERM
W0513 15:52:40.027000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7235 via signal SIGTERM
W0513 15:52:40.027000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7236 via signal SIGTERM
W0513 15:52:40.027000 7102 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 7237 via signal SIGTERM
Traceback (most recent call last):
File "/home/dpsk_a2a/DeepEP/tests/test_low_latency.py", line 172, in
torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 184, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 2 terminated with signal SIGSEGV
```
Node 2:
```
root@0540de603a8e:/home/dpsk_a2a/DeepEP# NCCL_DEBUG=WARN MASTER_ADDR= WORLD_SIZE=2 RANK=1 python3 tests/test_low_latency.py
Allocating buffer size: 2115.111296 MB ...
......
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 8 nranks 16 tag 4 - DONE
[rank 14] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.49 us, min_t=958.24 us, max_t=993.06 us
[rank 10] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.50 us, min_t=955.65 us, max_t=992.26 us
[rank 15] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.40 us, min_t=956.29 us, max_t=994.27 us
[rank 8] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.64 us, min_t=961.98 us, max_t=993.25 us
[rank 13] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.62 us, min_t=959.10 us, max_t=995.14 us
[rank 12] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.30 us, min_t=955.07 us, max_t=994.40 us
[rank 11] Dispatch + combine bandwidth: 22.65 GB/s, avg_t=973.62 us, min_t=961.54 us, max_t=991.52 us
[rank 9] Dispatch + combine bandwidth: 22.64 GB/s, avg_t=973.71 us, min_t=964.22 us, max_t=988.48 us
[rank 14] Dispatch bandwidth: 22.24 GB/s, avg_t=337.70 us | Combine bandwidth: 22.75 GB/s, avg_t=639.08 us
[rank 10] Dispatch bandwidth: 21.51 GB/s, avg_t=349.25 us | Combine bandwidth: 23.00 GB/s, avg_t=631.95 us
[rank 15] Dispatch bandwidth: 22.62 GB/s, avg_t=332.15 us | Combine bandwidth: 22.74 GB/s, avg_t=639.28 us
[rank 11] Dispatch bandwidth: 22.15 GB/s, avg_t=339.13 us | Combine bandwidth: 23.00 GB/s, avg_t=632.14 us
[rank 8] Dispatch bandwidth: 23.14 GB/s, avg_t=324.60 us | Combine bandwidth: 22.64 GB/s, avg_t=642.14 us
[rank 9] Dispatch bandwidth: 21.42 GB/s, avg_t=350.74 us | Combine bandwidth: 22.83 GB/s, avg_t=636.74 us
[rank 13] Dispatch bandwidth: 22.47 GB/s, avg_t=334.25 us | Combine bandwidth: 22.79 GB/s, avg_t=637.89 us
[rank 12] Dispatch bandwidth: 21.82 GB/s, avg_t=344.33 us | Combine bandwidth: 23.08 GB/s, avg_t=629.81 us
[rank 9] Dispatch send/recv time: 28.80 us | Combine send/recv time: 28.67 us
[rank 13] Dispatch send/recv time: 28.36 us | Combine send/recv time: 28.63 us
[rank 8] Dispatch send/recv time: 28.82 us | Combine send/recv time: 28.52 us
[rank 11] Dispatch send/recv time: 27.96 us | Combine send/recv time: 28.54 us
[rank 14] Dispatch send/recv time: 27.76 us | Combine send/recv time: 28.58 us
[rank 12] Dispatch send/recv time: 28.77 us | Combine send/recv time: 29.07 us
[rank 15] Dispatch send/recv time: 28.35 us | Combine send/recv time: 28.78 us
[rank 10] Dispatch send/recv time: 28.70 us | Combine send/recv time: 28.78 us
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 8 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 13 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 10 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 9 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 11 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 14 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 15 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:559: rank 12 nranks 16 tag 0 - ENTER
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 11 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 14 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 15 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 9 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 13 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 10 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 12 nranks 16 tag 4 - DONE
/home/dpsk_a2a/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 8 nranks 16 tag 4 - DONE
W0513 15:52:34.833000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5939 via signal SIGTERM
W0513 15:52:34.833000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5940 via signal SIGTERM
W0513 15:52:34.834000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5941 via signal SIGTERM
W0513 15:52:34.834000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5942 via signal SIGTERM
W0513 15:52:34.834000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5943 via signal SIGTERM
W0513 15:52:34.834000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5944 via signal SIGTERM
W0513 15:52:34.834000 5810 site-packages/torch/multiprocessing/spawn.py:160] Terminating process 5945 via signal SIGTERM
Traceback (most recent call last):
File "/home/dpsk_a2a/DeepEP/tests/test_low_latency.py", line 172, in
torch.multiprocessing.spawn(test_loop, args=(num_processes,), nprocs=num_processes)
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 328, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 284, in start_processes
while not context.join():
^^^^^^^^^^^^^^
File "/opt/ac2/lib/python3.12/site-packages/torch/multiprocessing/spawn.py", line 184, in join
raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGSEGV
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.