deepseek-ai / deepseek-ai/DeepEP
test_low_latency.py Failed on 2 node H800
- Dominant language
- Cuda
- Stars
- 10.1k
- Forks
- 1.4k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 2
Description
### Node1
```
MASTER_ADDR=WORLD_SIZE=2 RANK=0 python DeepEP/tests/test_low_latency.py
```
### Node2
```
MASTER_ADDR=WORLD_SIZE=2 RANK=1 python DeepEP/tests/test_low_latency.py
```
### Error message
```
/nvshmem/src/host/mem/mem_heap.cpp:1509: non-zero status: 2 cuMemCreate failed
/nvshmem/src/host/mem/mem_heap.cpp:1509: non-zero status: 2 cuMemCreate failed
/nvshmem/src/host/mem/mem_heap.cpp:1509: non-zero status: 2 cuMemCreate failed
/nvshmem/src/host/mem/mem_heap.cpp:1509: non-zero status: 2 cuMemCreate failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgress:59: socketProgress: Connection closed by remote peer e01-cn-e4v48e4dl07<46500>
/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:97: non-zero status: -6 /nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -6 /nvshmem/src/host/mem/mem_heap.cpp:940: non-zero status: -6 allgather of mem handles failed
/nvshmem/src/host/mem/mem_heap.cpp:1098: non-zero status: 7 register heap memory failed
/nvshmem/src/host/mem/mem_heap.cpp:1533: non-zero status: 7 register heap UC memory failed
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 172.18.0.177<43129> failed : Broken pipe
/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:91: non-zero status: -6 /nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /nvshmem/src/host/mem/mem_heap.cpp:940: non-zero status: -6 allgather of mem handles failed
/nvshmem/src/host/mem/mem_heap.cpp:1098: non-zero status: 7 register heap memory failed
/nvshmem/src/host/mem/mem_heap.cpp:1533: non-zero status: 7 register heap UC memory failed
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgressOpt:39: socketProgressOpt: Call to recv from 172.18.0.177<38567> failed : Broken pipe
/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:91: non-zero status: -6 /nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:496: non-zero status: -6 /nvshmem/src/host/mem/mem_heap.cpp:940: non-zero status: -6 allgather of mem handles failed
/nvshmem/src/host/mem/mem_heap.cpp:1098: non-zero status: 7 register heap memory failed
/nvshmem/src/host/mem/mem_heap.cpp:1533: non-zero status: 7 register heap UC memory failed
/nvshmem/src/host/mem/mem_heap.cpp:532: non-zero status: 1 cuMemAddressFree failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/host/mem/mem_heap.cpp:532: non-zero status: 1 cuMemAddressFree failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/host/mem/mem_heap.cpp:532: non-zero status: 1 cuMemAddressFree failed
/nvshmem/src/host/mem/mem_heap.cpp:1590: non-zero status: 7 allocate_physical_memory_to_heap failed
/nvshmem/src/modules/bootstrap/uid/ncclSocket/ncclsocket_socket.cpp:socketProgress:59: socketProgress: Connection closed by remote peer e01-cn-e4v48e4dl06<41542>
/nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:97: non-zero status: -6 /nvshmem/src/modules/bootstrap/uid/bootstrap_uid.cpp:499: non-zero status: -6 /nvshmem/src/host/mem/mem_heap.cpp:940: non-zero status: -6 allgather of mem handles failed
/nvshmem/src/host/mem/mem_heap.cpp:1098: non-zero status: 7 register heap memory failed
/nvshmem/src/host/mem/mem_heap.cpp:1533: non-zero status: 7 register heap UC memory failed
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
[/nvshmem/src/host/coll/barrier/barrier.cpp:21] cuda failed with an illegal memory access was encountered
/nvshmem/src/host/util/cs.cpp:21: non-zero status: 16: Device or resource busy, exiting... mutex destroy failed
/nvshmem/src/host/mem/mem_heap.cpp:532: non-zero status: 1 cuMemAddressFree failed
```
How can I fix this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.