microsoft / microsoft/TRELLIS.2
illegal memory access in multi-gpu training
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.3k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
Thanks for your great work! I encounter an issue when I use multi-gpu training. Single-gpu training works fine and no error. After the network initialization, I got:
[rank1]:[E115 15:39:38.894064351 ProcessGroupNCCL.cpp:1899] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x798ee03785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x798ee030d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x798ee074f422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x798e758e55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x798e758f5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x798e758f73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x798e758f8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x798e65aea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x798ee129caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x798ee1329c6c in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x798ee03785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x798ee030d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x798ee074f422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x798e758e55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x798e758f5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x798e758f73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x798e758f8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x798e65aea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x798ee129caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x798ee1329c6c in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x798ee03785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xcc7b9e (0x798e758c7b9e in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x9165ed (0x798e755165ed in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdf0e6 (0x798e65aea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #4: <unknown function> + 0x9caa4 (0x798ee129caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x129c6c (0x798ee1329c6c in /lib/x86_64-linux-gnu/libc.so.6)
[rank3]:[E115 15:39:38.012111423 ProcessGroupNCCL.cpp:1899] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e892f9785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7e892f90d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7e892fdeb422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7e88c4ee55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7e88c4ef5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7e88c4ef73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7e88c4ef8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7e88b50ea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x7e893089caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x7e8930929c6c in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
[rank2]:[E115 15:39:38.012685430 ProcessGroupNCCL.cpp:1899] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7d9ab0f785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7d9ab0f0d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7d9abd7a5422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7d9a52ce55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7d9a52cf5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7d9a52cf73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7d9a52cf8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7d9a42eea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x7d9abe49caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x7d9abe529c6c in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7d9ab0f785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7d9ab0f0d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7d9abd7a5422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7d9a52ce55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7d9a52cf5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7d9a52cf73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7d9a52cf8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7d9a42eea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x7d9abe49caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x7d9abe529c6c in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7d9ab0f785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xcc7b9e (0x7d9a52cc7b9e in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x9165ed (0x7d9a529165ed in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdf0e6 (0x7d9a42eea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #4: <unknown function> + 0x9caa4 (0x7d9abe49caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x129c6c (0x7d9abe529c6c in /lib/x86_64-linux-gnu/libc.so.6)
[rank0]:[E115 15:39:38.016628941 ProcessGroupNCCL.cpp:1899] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7983615785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x79836150d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x798361967422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7982f6ae55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7982f6af5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7982f6af73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7982f6af8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7982e6cea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x79836249caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x798362529c6c in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7983615785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x79836150d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x798361967422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7982f6ae55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7982f6af5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7982f6af73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7982f6af8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7982e6cea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x79836249caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x798362529c6c in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7983615785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xcc7b9e (0x7982f6ac7b9e in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x9165ed (0x7982f67165ed in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdf0e6 (0x7982e6cea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #4: <unknown function> + 0x9caa4 (0x79836249caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x129c6c (0x798362529c6c in /lib/x86_64-linux-gnu/libc.so.6)
what(): [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e892f9785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x7e892f90d4a2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7e892fdeb422 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7e88c4ee55a6 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0x70 (0x7e88c4ef5840 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x782 (0x7e88c4ef73d2 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7e88c4ef8fdd in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xdf0e6 (0x7e88b50ea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #8: <unknown function> + 0x9caa4 (0x7e893089caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x129c6c (0x7e8930929c6c in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1905 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7e892f9785e8 in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0xcc7b9e (0x7e88c4ec7b9e in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x9165ed (0x7e88c4b165ed in /home/hongchix/miniconda3/envs/trellis/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xdf0e6 (0x7e88b50ea0e6 in /home/hongchix/miniconda3/envs/trellis/bin/../lib/libstdc++.so.6)
frame #4: <unknown function> + 0x9caa4 (0x7e893089caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #5: <unknown function> + 0x129c6c (0x7e8930929c6c in /lib/x86_64-linux-gnu/libc.so.6)
W0115 15:39:39.059000 3585642 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 3586807 via signal SIGTERM
W0115 15:39:39.060000 3585642 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 3586809 via signal SIGTERM
W0115 15:39:39.060000 3585642 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 3586810 via signal SIGTERM
Error: process 1 terminated with signal SIGABRT
Do you know the reason of this issue? Thank you for your time.
My env: Cuda 12.8 Ubuntu GPU: NVIDIA RTX PRO 6000 Blackwell
Best,
Hongchi
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the multi-GPU training failure after network initialization, comparing it with the working single-GPU case. Run with CUDA_LAUNCH_BLOCKING=1 and consider TORCH_USE_CUDA_DSA to narrow the illegal memory access; done means identifying the failing project component and demonstrating that multi-GPU training completes without the NCCL watchdog error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100