[Bug] RuntimeError: CUDA error: an illegal memory access was encountered

Open

#3,258 opened on Mar 14, 2025 14 comments 0 reactions 1 assignee Claimed by @grimoire View on GitHub

Dominant language: Python
Stars: 8.1k
Forks: 748
Avg merge: 6d 2h
Merged PRs (30d): 54

Description

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

RuntimeError: CUDA error: an illegal memory access was encountered
我发现这个错误在多人调用接口的时候，容易发生，接口调用代码见【reproduction】
运行环境（nvidia-smi显示的显卡驱动 550.144.03； cuda驱动 CUDA Version: 12.4）：
```
Package Version Editable project location
--------------------------------- ------------- -------------------------
accelerate 1.4.0
addict 2.4.0
aiohappyeyeballs 2.4.6
aiohttp 3.11.12
aiohttp-cors 0.7.0
aiosignal 1.3.2
airportsdata 20241001
annotated-types 0.7.0
anthropic 0.46.0
anyio 4.8.0
astor 0.8.1
asttokens 3.0.0
async-timeout 5.0.1
attrs 25.1.0
baidu-aip 4.16.13
bcrypt 4.3.0
beautifulsoup4 4.13.3
bitsandbytes 0.45.3
blake3 1.0.4
blinker 1.9.0
Brotli 1.1.0
cachetools 5.5.2
certifi 2025.1.31
cffi 1.17.1
cfgv 3.4.0
chardet 5.2.0
charset-normalizer 3.4.1
click 8.1.8
cloudpickle 3.1.1
colorful 0.5.6
colossalai 0.4.9
compressed-tensors 0.9.1
contexttimer 0.3.3
contourpy 1.3.1
cryptography 44.0.2
cuda-bindings 12.8.0
cuda-python 12.8.0
cycler 0.12.1
datasets 3.3.2
decorator 5.1.1
decord 0.6.0
deepspeed 0.15.4
Deprecated 1.2.18
depyf 0.18.0
diffusers 0.29.0
dill 0.3.8
diskcache 5.6.3
distlib 0.3.9
distro 1.9.0
docstring_parser 0.16
einops 0.8.1
exceptiongroup 1.2.2
executing 2.2.0
fabric 3.2.2
fastapi 0.115.8
filelock 3.17.0
fire 0.7.0
flashinfer-python 0.2.1.post2
Flask 3.1.0
Flask-Cors 5.0.0
fonttools 4.56.0
frozenlist 1.5.0
fsspec 2024.12.0
galore-torch 1.0
gevent 24.11.1
gguf 0.10.0
gmpy2 2.1.5
google 3.0.0
google-api-core 2.24.1
google-auth 2.38.0
googleapis-common-protos 1.68.0
greenlet 3.1.1
grpcio 1.70.0
h11 0.14.0
h2 4.2.0
hf_transfer 0.1.9
hjson 3.1.0
hpack 4.1.0
httpcore 1.0.7
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.29.1
hyperframe 6.1.0
identify 2.6.9
idna 3.10
importlib_metadata 8.6.1
iniconfig 2.0.0
interegular 0.3.3
invoke 2.2.0
ipdb 0.13.13
ipython 8.32.0
itsdangerous 2.2.0
jedi 0.19.2
Jinja2 3.1.5
jiter 0.8.2
jsonschema 4.23.0
jsonschema-specifications 2024.10.1
kiwisolver 1.4.8
lark 1.2.2
litellm 1.61.13
llvmlite 0.44.0
lm-format-enforcer 0.10.10
lmdeploy 0.6.5 /disk2/elivate/lmdeploy
loguru 0.7.3
markdown-it-py 3.0.0
MarkupSafe 3.0.2
matplotlib 3.10.0
matplotlib-inline 0.1.7
mdurl 0.1.2
mistral_common 1.5.3
mmengine-lite 0.10.6
modelscope 1.23.1
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.1.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.3
nodeenv 1.9.1
numba 0.61.0
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-ml-py 12.570.86
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
openai 1.63.2
opencensus 0.11.4
opencensus-context 0.1.3
opencv-python-headless 4.11.0.86
orjson 3.10.15
outlines 0.1.11
outlines_core 0.1.26
packaging 24.2
pandas 2.2.3
paramiko 3.5.1
parso 0.8.4
partial-json-parser 0.2.1.1.post5
peft 0.11.1
pexpect 4.9.0
pillow 11.1.0
pip 25.0.1
platformdirs 4.3.6
pluggy 1.5.0
plumbum 1.9.0
pre_commit 4.1.0
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.0.2
prompt_toolkit 3.0.50
propcache 0.3.0
proto-plus 1.26.0
protobuf 5.29.3
psutil 7.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.0
pyairports 2.1.1
pyarrow 19.0.1
pyasn1 0.6.1
pyasn1_modules 0.4.1
pybind11 2.13.6
pycountry 24.6.1
pycparser 2.22
pydantic 2.10.6
pydantic_core 2.27.2
Pygments 2.19.1
PyNaCl 1.5.0
pynvml 12.0.0
pyparsing 3.2.1
PySocks 1.7.1
pytest 8.3.4
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.20
pytz 2025.1
PyYAML 6.0.2
pyzmq 26.2.1
ray 2.42.1
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 13.9.4
rpds-py 0.23.1
rpyc 6.0.0
rsa 4.9
safetensors 0.4.5
sentencepiece 0.2.0
setproctitle 1.3.4
setuptools 75.8.0
sgl-kernel 0.0.3.post6
shortuuid 1.0.13
shtab 1.7.1
six 1.17.0
smart-open 7.1.0
sniffio 1.3.1
soupsieve 2.6
stack-data 0.6.3
starlette 0.45.3
sympy 1.13.1
termcolor 2.5.0
tiktoken 0.9.0
tokenizers 0.20.3
tomli 2.2.1
torch 2.5.1
torchao 0.8.0
torchaudio 2.5.1
torchvision 0.20.1
tqdm 4.67.1
traitlets 5.14.3
transformers 4.46.3
triton 3.0.0
trl 0.8.6
typeguard 4.4.2
typing_extensions 4.12.2
tyro 0.9.16
tzdata 2025.1
urllib3 2.3.0
uvicorn 0.29.0
uvloop 0.21.0
virtualenv 20.29.2
vllm 0.7.2
watchfiles 1.0.4
wcwidth 0.2.13
websockets 15.0
Werkzeug 3.1.3
wheel 0.45.1
wrapt 1.17.2
xformers 0.0.28.post3
xgrammar 0.1.10
xxhash 3.5.0
yapf 0.43.0
yarl 1.18.3
zipp 3.21.0
zope.event 5.0
zope.interface 7.2
zstandard 0.19.0
```

### Reproduction

运行命令：
```
lmdeploy serve api_server /disk2/elivate/DeepSeek/DeepSeek-R1 --tp 8 --backend pytorch --chat-template deepseek \
--cache-max-entry-count 0.5 --server-name 0.0.0.0 --server-port 23333
```
接口调用代码：
```
#coding:utf-8
from openai import OpenAI
import httpx
#运行代码和服务器是一个ip时，需要讲0.0.0.0改成服务端ip（端口号23333可能也需要改成指定的端口）
client = OpenAI(
api_key='YOUR_API_KEY',
base_url="http://0.0.0.0:23333/v1",
http_client=httpx.Client(verify=False)
)
model_name = client.models.list().data[0].id

input_format = '''
任何输出都要有思考过程，输出内容必须以 "\n\n嗯" 开头。仔细揣摩用户意图，之后提供逻辑清晰且内容完整的回答，可以使用Markdown格式优化信息呈现。\n\n

{}'''
#是否深度思考（prompt强制输出思考内容）
use_think=True
input_str = '输入'
if use_think:
cur_message = [{"role": "user", "content": input_format.format(input_str)}]
else:
cur_message = [{"role": "user", "content": input_str}]

# #流式输出（视觉效果好）
# response = client.chat.completions.create(
# model=model_name,
# messages=cur_message,
# temperature=0.8,
# top_p=0.8,
# stream=True,
# )
# text=''
# for chunk in response:
# # 判断回复是否非空
# if chunk.choices[0].delta.content:
# text+=chunk.choices[0].delta.content
# print(chunk.choices[0].delta.content, end='') # 设置 end='' 实现不换行，视觉上拼接输出
# #打印
# print('\n',text)

#非流式输出
response = client.chat.completions.create(
model=model_name,
messages=cur_message,
temperature=0.6,
top_p=0.8,
)
#打印
print(response.choices[0].message.content)

```

### Environment

```Shell
/disk2/eliviate/lmdeploy/lmdeploy/cli/entrypoint.py
sys.platform: linux
Python: 3.10.16 | packaged by conda-forge | (main, Dec 5 2024, 14:16:10) [GCC 13.3.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1,2,3,4,5,6,7: NVIDIA H200
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: gcc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
PyTorch: 2.5.1+cu124
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 12.4
- NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 90.1
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.20.1+cu124
LMDeploy: 0.6.5+
transformers: 4.46.3
gradio: Not Found
fastapi: 0.115.8
pydantic: 2.10.6
triton: 3.0.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX PIX SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX PIX SYS SYS SYS SYS 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX PIX NODE NODE 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS PIX PIX NODE NODE 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE PIX PIX 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE PIX PIX 48-95,144-191 1 N/A
NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE NODE SYS SYS SYS SYS
NIC1 NODE NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE NODE SYS SYS SYS SYS
NIC2 PIX PIX NODE NODE SYS SYS SYS SYS NODE NODE X PIX NODE NODE SYS SYS SYS SYS
NIC3 PIX PIX NODE NODE SYS SYS SYS SYS NODE NODE PIX X NODE NODE SYS SYS SYS SYS
NIC4 NODE NODE PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE X PIX SYS SYS SYS SYS
NIC5 NODE NODE PIX PIX SYS SYS SYS SYS NODE NODE NODE NODE PIX X SYS SYS SYS SYS
NIC6 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS X PIX NODE NODE
NIC7 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS SYS PIX X NODE NODE
NIC8 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS NODE NODE X PIX
NIC9 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS SYS NODE NODE PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
```

### Error traceback

```Shell
错误输出：

Traceback (most recent call last):
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 482, in _start_tp_process
func(rank, *args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 439, in _tp_model_loop
model_forward(
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 156, in model_forward
output = model(**input_dict)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/backends/cuda/graph_runner.py", line 149, in __call__
return self.model(**kwargs)
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 702, in forward
hidden_states = self.model(
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 654, in forward
hidden_states, residual = decoder_layer(
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 555, in forward
hidden_states = self.self_attn(
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/disk2/eliviate/lmdeploy/lmdeploy/pytorch/models/deepseek_v2.py", line 256, in forward
query_states[..., nope_size:] = q_pe
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[rank6]:[E314 14:33:46.584848437 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 6] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f0f0116c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f0f011166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f0f01534a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f0eb7025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f0eb702a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f0eb7031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f0eb703361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f0f01c4a5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f0f02370ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f0f02402850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank4]:[E314 14:33:46.585238329 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 4] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5e2fb6c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f5e2fb166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f5e2ffd4a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5de5a25726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f5de5a2a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f5de5a31b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f5de5a3361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f5e306ea5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f5e30e10ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f5e30ea2850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 6] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

what(): [PG ID 0 PG GUID 0(default_pg) Rank 4] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5e2fb6c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f5e2fb166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f5e2ffd4a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f5de5a25726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f5de5a2a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f5de5a31b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f5de5a3361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f5e306ea5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f5e30e10ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f5e30ea2850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank2]:[E314 14:33:46.588302933 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4c560b9446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f4c560636e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f4c561a5a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f4c0c025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f4c0c02a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f4c0c031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f4c0c03361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f4c56b785c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f4c5729eac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f4c57330850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank5]:[E314 14:33:46.588801693 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f581a0b9446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f581a0636e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f581a1a5a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f57d0025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f57d002a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f57d0031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f57d003361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f581abcf5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f581b2f5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f581b387850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank3]:[E314 14:33:46.588877826 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff69c16c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7ff69c1166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7ff69c544a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7ff652025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7ff65202a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7ff652031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7ff65203361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7ff69cc5a5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7ff69d380ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7ff69d412850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
terminate called after throwing an instance of 'c10::DistBackendError'
[rank1]:[E314 14:33:46.589305831 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd56276c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fd5627166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fd562b73a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fd518625726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fd51862a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fd518631b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd51863361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7fd5632895c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7fd5639afac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7fd563a41850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
[rank7]:[E314 14:33:46.589658386 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 7] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fa19536c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fa1953166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fa19578ca18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fa14b225726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fa14b22a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fa14b231b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fa14b23361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7fa195ea25c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7fa1965c8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7fa19665a850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4c560b9446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f4c560636e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f4c561a5a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f4c0c025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f4c0c02a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f4c0c031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f4c0c03361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f4c56b785c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f4c5729eac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f4c57330850 in /lib/x86_64-linux-gnu/libc.so.6)

what(): [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f581a0b9446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f581a0636e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f581a1a5a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f57d0025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f57d002a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f57d0031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f57d003361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f581abcf5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f581b2f5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f581b387850 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f581a0b9446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7f57cfca071b in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7f581abcf5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7f581b2f5ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126850 (0x7f581b387850 in /lib/x86_64-linux-gnu/libc.so.6)
what():
[PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff69c16c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7ff69c1166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7ff69c544a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7ff652025726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7ff65202a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7ff652031b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7ff65203361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7ff69cc5a5c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7ff69d380ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7ff69d412850 in /lib/x86_64-linux-gnu/libc.so.6)

what(): [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd56276c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fd5627166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fd562b73a18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fd518625726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fd51862a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fd518631b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd51863361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7fd5632895c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7fd5639afac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7fd563a41850 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd56276c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fd5182a071b in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fd5632895c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7fd5639afac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126850 (0x7fd563a41850 in /lib/x86_64-linux-gnu/libc.so.6)
what():
[PG ID 0 PG GUID 0(default_pg) Rank 7] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fa19536c446 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fa1953166e4 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fa19578ca18 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fa14b225726 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fa14b22a3f0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fa14b231b5a in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fa14b23361d in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7fa195ea25c0 in /disk2/condaenvs/deepseek/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7fa1965c8ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7fa19665a850 in /lib/x86_64-linux-gnu/libc.so.6)

/disk2/condaenvs/deepseek/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

[Bug] RuntimeError: CUDA error: an illegal memory access was encountered

Description

Contributor guide

Assessment

Get new issues in your inbox