facebookresearch / facebookresearch/SlowFast

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Open
#376 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.4k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Trying to train on my own dataset on google colab, however I can't get past this cuda error. Any help on this?

Traceback (most recent call last):
File "tools/run_net.py", line 44, in
main()
File "tools/run_net.py", line 25, in main
launch_job(cfg=cfg, init_method=args.init_method, func=train)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 297, in launch_job
func(cfg=cfg)
File "/content/gdrive/My Drive/SlowFast/tools/train_net.py", line 383, in train
misc.log_model_info(model, cfg, use_train_input=True)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 174, in log_model_info
get_model_stats(model, cfg, "flop", use_train_input)
File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 152, in get_model_stats
count_dict, *_ = model_stats_fun(model, inputs)
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/flop_count.py", line 130, in flop_count
for op, flop in flop_counter.by_operator().items():
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 239, in by_operator
stats = self._analyze()
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 494, in _analyze
graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases)
File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 161, in _get_scoped_trace_graph
graph, _ = _get_trace_graph(module, inputs)
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 1139, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 130, in forward
self._force_outplace,
File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 116, in wrapper
outs.append(self.inner(*trace_inputs))
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/video_model_builder.py", line 588, in forward
x = self.s1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 114, in forward
x[pathway] = m(x[pathway])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 190, in forward
x = self.conv(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
result = self._slow_forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 521, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.