deepseek-ai / deepseek-ai/DeepEP

"RuntimeError: Failed: CUDA error /opt/DeepEP/csrc/kernels/internode.cu:133 ' named symbol not found" in container when running test_intranode.py

Open
#87 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hi all,

I have installed gdrcopy and nvshmem in container (with gdrcopy_copybw, nvshmem-info -a verified), but when I run

`python3 tests/test_intranode.py`

I encounter the following errors:

```
[config] num_tokens=4096, hidden=7168, num_topk=8
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
terminate called after throwing an instance of 'EPException'
what(): Failed: CUDA error /opt/DeepEP/csrc/kernels/runtime.cu:25 'named symbol not found'
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79673 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79674 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79675 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79676 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79678 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79679 via signal SIGTERM
W0324 10:50:24.281000 139655644297024 torch/multiprocessing/spawn.py:145] Terminating process 79680 via signal SIGTERM
Traceback (most recent call last):
File "/opt/DeepEP/tests/test_intranode.py", line 225, in
torch.multiprocessing.spawn(test_loop, args=(num_processes, ), nprocs=num_processes)
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 281, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 237, in start_processes
while not context.join():
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 188, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 75, in _wrap
fn(i, *args)
File "/opt/DeepEP/tests/test_intranode.py", line 213, in test_loop
test_main(i, local_rank, num_ranks, rank, buffer, group)
File "/opt/DeepEP/tests/test_intranode.py", line 58, in test_main
buffer.get_dispatch_layout(topk_idx, num_experts)
File "/usr/local/lib/python3.10/dist-packages/deep_ep-1.0.0+c4b8ffc-py3.10-linux-x86_64.egg/deep_ep/buffer.py", line 227, in get_dispatch_layout
self.runtime.get_dispatch_layout(topk_idx, num_experts, getattr(previous_event, 'event', None),
RuntimeError: Failed: CUDA error /opt/DeepEP/csrc/kernels/internode.cu:133 'named symbol not found'
```

Any hints on how to solve the problem?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.