facebookresearch / facebookresearch/SlowFast
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Trying to train on my own dataset on google colab, however I can't get past this cuda error. Any help on this?
Traceback (most recent call last):
File "tools/run_net.py", line 44, in
main()
File "tools/run_net.py", line 25, in main
launch_job(cfg=cfg, init_method=args.init_method, func=train)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 297, in launch_job
func(cfg=cfg)
File "/content/gdrive/My Drive/SlowFast/tools/train_net.py", line 383, in train
misc.log_model_info(model, cfg, use_train_input=True)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 174, in log_model_info
get_model_stats(model, cfg, "flop", use_train_input)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 152, in get_model_stats
count_dict, *_ = model_stats_fun(model, inputs)
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/flop_count.py", line 130, in flop_count
for op, flop in flop_counter.by_operator().items():
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 239, in by_operator
stats = self._analyze()
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 494, in _analyze
graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases)
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 161, in _get_scoped_trace_graph
graph, _ = _get_trace_graph(module, inputs)
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 1139, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 130, in forward
self._force_outplace,
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 116, in wrapper
outs.append(self.inner(*trace_inputs))
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/video_model_builder.py", line 588, in forward
x = self.s1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 114, in forward
x[pathway] = m(x[pathway])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 190, in forward
x = self.conv(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 521, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
Contributor guide
Assessment
This issue has not been assessed yet.