The problem of slow GPU inference speed in exporting onnx models

Dear Facebook Development Team
I saw this description in the guide: The converted model is able to run in either Python or C++ without detectron2/torchvision dependency, on CPU or GPUs. It has a runtime optimized for CPU & mobile inference, but not optimized for GPU inference. After actual testing, I found that the CPU time is only one-third of the GPU time. Accelerated optimization will significantly improve inference speed. Therefore, I am wondering if it is possible to add optimization to the exported onnx GPU inference, as this is also a fundamental feature that is universal in the project.


Start by reproducing the reported CPU and GPU timing for an exported ONNX model using the information in the issue and its attached image. Investigate the exported-model inference path and define done as a measurable improvement in GPU inference speed without losing the stated Python and C++ execution support.

Written by the indexing model from the issue text.