deepseek-ai / deepseek-ai/DeepEP

[DeepEPv2] test_agrs.py failed due to CUDA_ERROR_NOT_SUPPORTED, operation not supported

Open
#656 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

Hi,

we tested the test_agrs.py and found it will be failed on H20:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap
fn(i, *args)
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/DeepEP/tests/elastic/test_agrs.py", line 125, in test
buffer.destroy_agrs_session()
File "/usr/local/lib/python3.12/dist-packages/deep_ep/buffers/elastic.py", line 503, in destroy_agrs_session
self.runtime.destroy_agrs_session()
RuntimeError: CUDA driver exception (/workspace/DeepEP/csrc/kernels/backend/cuda_driver.cu:51): 801 (CUDA_ERROR_NOT_SUPPORTED, operation not supported)

https://github.com/deepseek-ai/DeepEP/blob/d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb/csrc/kernels/backend/cuda_driver.cu#L51

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.