RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

When I run this command:
python tools/run_net.py --cfg configs/AVA/c2/SLOWFAST_32x2_R101_50_50_v2.1.yaml DATA.PATH_TO_DATA_DIR /home/viplab2/Documents/AVA TEST.CHECKPOINT_FILE_PATH /home/viplab2/Documents/zhengchenghuan/slowfast/ava_checkpoint/SLOWFAST_32x2_R101_50_50_v2.1.pkl TRAIN.ENABLE False TEST.CHECKPOINT_TYPE caffe2

I got a error:

[05/15 10:25:53][INFO] slowfast.datasets.ava_helper: 62: Finished loading image paths from: /home/viplab2/Documents/AVA/frame_lists/val.csv
Traceback (most recent call last):
File "tools/run_net.py", line 65, in
main()
File "tools/run_net.py", line 57, in main
daemon=False,
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
while not context.join():
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
raise Exception(msg)
Exception:

-- Process 2 terminated with the following error:
Traceback (most recent call last):
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/slowfast/utils/multiprocessing.py", line 51, in run
func(cfg)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/tools/test_net.py", line 187, in test
perform_test(test_loader, model, test_meter, cfg)
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
return func(*args, **kwargs)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/tools/test_net.py", line 63, in perform_test
preds = model(inputs, meta["boxes"])
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 445, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/slowfast/models/video_model_builder.py", line 387, in forward
x = self.s4(x)
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/slowfast/models/resnet_helper.py", line 553, in forward
x = nln(x)
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/viplab2/Documents/zhengchenghuan/slowfast/slowfast/models/nonlocal_helper.py", line 123, in forward
theta_phi = torch.einsum("nct,ncp->ntp", (theta, phi))
File "/home/viplab2/anaconda3/lib/python3.7/site-packages/torch/functional.py", line 292, in einsum
return _VF.einsum(equation, operands)
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

I don't know how to fix it. Can anybody help me? Thank you.