dmlc / dmlc/decord

About torchrun startup,This problem occurs when I use torchrun multi-card training

Open
#327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
232
PR merge metrics
No merged PRs in 30d

Description

File "/mnt/code3/lxd/ittvideo/testdataset.py", line 924, in __getitem__
vr = decord.VideoReader(path, width=self.width, height=self.height,ctx=self.ctx)
File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/video_reader.py", line 54, in __init__
self._handle = _CAPI_VideoReaderGetVideoReader(
File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/_ffi/_ctypes/function.py", line 173, in __call__ check_call(_LIB.DECORDFuncCall( File "/home/ps/.local/lib/python3.9/site-packages/decord-0.6.0-py3.9-linux-x86_64.egg/decord/_ffi/base.py", line 78, in check_call raise DECORDError(err_str) decord._ffi.base.DECORDError: [12:18:41] /mnt/code3/lxd/decord/src/video/nvcodec/cuda_stream.cc:20: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: initialization error at line: 20 in file: /mnt/code3/lxd/decord/src/video/nvcodec/cuda_stream.cc

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported call in testdataset.py at line 924 and trace through decord/video_reader.py to nvcodec/cuda_stream.cc at line 20. Reproduce the failure with torchrun multi-card training and establish a reliable CUDA initialization failure case; done means the issue has a documented cause and a verified resolution or clear reproduction details.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.