About torchrun startup,This problem occurs when I use torchrun multi-card training
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 232
- PR merge metrics
- No merged PRs in 30d
Description
File "/mnt/code3/lxd/ittvideo/testdataset.py", line 924, in __getitem__
vr = decord.VideoReader(path, width=self.width, height=self.height,ctx=self.ctx)
File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/video_reader.py", line 54, in __init__
self._handle = _CAPI_VideoReaderGetVideoReader(
File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/_ffi/_ctypes/function.py", line 173, in __call__ check_call(_LIB.DECORDFuncCall( File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/_ffi/base.py", line 78, in check_call raise DECORDError(err_str) decord._ffi.base.DECORDError: [12:18:41] /mnt/code3/lxd/decord/src/video/nvcodec/cuda_stream.cc:20: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error at line: 20 in file: /mnt/code3/lxd/decord/src/video/nvcodec/cuda_stream.cc
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported call in testdataset.py at line 924 and trace through decord/video_reader.py to nvcodec/cuda_stream.cc at line 20. Reproduce the failure with torchrun multi-card training and establish a reliable CUDA initialization failure case; done means the issue has a documented cause and a verified resolution or clear reproduction details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100