Weird error: "RuntimeError: Error while calling cudnnConvolutionForward ... code: 7, reason: A call to cuDNN failed"

I tried another face detection method which returns bounding boxes values of float type. Then I convert them with int() and feed them to face_recognition.face_encodings() (which I assign as get_face_encoding()). Then I get error detailed as below:

Traceback (most recent call last):
File "predict_tensorrt_video.py", line 665, in
main()
File "predict_tensorrt_video.py", line 88, in inner
retval = fnc(*args, **kwargs)
File "predict_tensorrt_video.py", line 659, in main
run_inference(args.video_in, args.video_out, candidate_id, current_time)
File "predict_tensorrt_video.py", line 549, in run_inference
face_encoding = get_face_encodings(frame, css_type_face_location, 0)[0]
File "/home/gate/.virtualenvs/lffd/lib/python3.6/site-packages/face_recognition/api.py", line 210, in face_encodings
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
File "/home/gate/.virtualenvs/lffd/lib/python3.6/site-packages/face_recognition/api.py", line 210, in
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
RuntimeError: Error while calling cudnnConvolutionForward( context(), &alpha, descriptor(data), data.device(), (const cudnnFilterDescriptor_t)filter_handle, filters.device(), (const cudnnConvolutionDescriptor_t)conv_handle, (cudnnConvolutionFwdAlgo_t)forward_algo, forward_workspace, forward_workspace_size_in_bytes, &beta, descriptor(output), output.device()) in file /home/gate/dlib-19.17/dlib/cuda/cudnn_dlibapi.cpp:1007. code: 7, reason: A call to cuDNN failed
cudaStreamDestroy() failed. Reason: invalid device ordinal
cudaFree() failed. Reason: invalid device pointer
cudaFreeHost() failed. Reason: invalid argument
cudaStreamDestroy() failed. Reason: unknown error
cudaFree() failed. Reason: invalid device pointer
cudaFreeHost() failed. Reason: invalid argument
cudaFree() failed. Reason: invalid device pointer
Segmentation fault (core dumped)

Anyone knows how to solve it? I've struggled with it all day. Thank you a lot!!

Start with predict_tensorrt_video.py at run_inference line 549 and face_recognition/api.py at line 210; reproduce the failure using the reported face locations and inspect the CUDA/cuDNN errors in the traceback. Done means identifying a reproducible cause and documenting or applying a verified fix without the cleanup errors or segmentation fault.