deepinsight / deepinsight/insightface

GPU Multithread report errors :CUDA failure 700

Open
#2,302 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

GPU Multithreading always cause this problem:
CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=*** ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyDeviceToHost, static_cast(stream.GetHandle())); and
CUDNN failure 8: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=***; expr=cudnnConvolutionForward(cudnn_handle, &alpha, s_.x_tensor, s_.x_data, s_.w_desc, s_.w_data, s_.conv_desc, s_.algo, workspace.get(), s_.workspace_bytes, &beta, s_.y_tensor, s_.y_data); and
CUDNN failure 4
But gpu single threading has no problem

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.