ONNX export of Faster R-CNN MobileNet V3 fails in ONNX Runtime

🐛 Describe the bug

Hi, I'm trying to export Faster R-CNN with MobileNet V3 backbone to ONNX, but have encountered several problems.

1. When I feed an image containing only a single object such as this, ONNX Runtime raises the following error:

Traceback (most recent call last):
  File "test_onnx.py", line 51, in <module>
    ort_output = ort_session.run(None, ort_input)
  File "/usr/local/anaconda/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 188, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_1537' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:36 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&, bool) size != 0 && (input_shape.Size() % size) == 0 was false. The input tensor cannot be reshaped to the requested shape. Input shape:{71,363}, requested shape:{-1,4}

2. Feeding images with multiple objects such as this causes no errors. However, it seems the results between PyTorch and ONNX Runtime are not consistent.

PyTorch scores:  tensor([0.9987, 0.9986, 0.8645, 0.2566, 0.1377, 0.0731, 0.0546])

ORT scores:  [0.9980155  0.99794    0.9972451  0.9964805  0.99622935 0.995884
 0.99581414 0.99375397 0.98647714 0.9856754  0.62823844 0.42497706
 0.20502456 0.16712733 0.07833275 0.07327942 0.06243803 0.05234617]

Here is a script to reproduce the issues.

import io
import requests

import numpy as np
import onnx
import onnxruntime
from PIL import Image
import torch
import torchvision
import torchvision.transforms.functional as TF


if __name__ == '__main__':
    # Load pretrained model
    torch_model = torchvision.models.detection.fasterrcnn_mobilenet_v3_large_fpn(pretrained=True)
    torch_model.eval()

    # Export to ONNX
    dummy_input = torch.randn(1, 3, 300, 300)
    onnx_model_path = './fasterrcnn_mobilenet_v3.onnx'
    
    torch.onnx.export(torch_model, dummy_input, onnx_model_path,
                      input_names=['images'],
                      output_names=['boxes', 'labels', 'scores'],
                      do_constant_folding=True,
                      export_params=True, opset_version=11, verbose=True)
    
    # Test the ONNX model with ONNX Runtime
    ort_session = onnxruntime.InferenceSession(onnx_model_path)

    def to_numpy(tensor):
        return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()
    
    with torch.no_grad():
        # Load an image with only single object
        image = Image.open(io.BytesIO(requests.get('https://farm8.staticflickr.com/7178/6859626097_3b561d69bf_z.jpg').content))
        
        # Image with multiple objects: no error, but produces inconsistent results
        # image = Image.open(io.BytesIO(requests.get('https://farm4.staticflickr.com/3334/3253284080_3268267cc5_z.jpg').content))
        
        image = image.resize((300, 300))
        torch_input = torch.unsqueeze(TF.to_tensor(image), 0)

        # PyTorch works fine
        torch_output = torch_model(torch_input)

        print('PyTorch scores: ', torch_output[0]['scores'])

        # ONNX Runtime raises an error or produces inconsistent results
        ort_input = {ort_session.get_inputs()[0].name: to_numpy(torch_input)}
        ort_output = ort_session.run(None, ort_input)

        print('ORT scores: ', ort_output[2])

Versions

PyTorch version: 1.9.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.27

Python version: 3.8.8 (default, Apr 13 2021, 19:58:26) [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Tesla V100-DGXS-32GB
GPU 1: Tesla V100-DGXS-32GB
GPU 2: Tesla V100-DGXS-32GB
GPU 3: Tesla V100-DGXS-32GB

Nvidia driver version: 450.142.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.20.1
[pip3] numpydoc==1.1.0
[pip3] torch==1.9.1
[pip3] torchvision==0.10.1
[conda] blas 1.0 mkl
[conda] cudatoolkit 10.2.89 hfd86e86_1
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2021.2.0 h06a4308_296
[conda] mkl-service 2.3.0 py38h27cfd23_1
[conda] mkl_fft 1.3.0 py38h42c9631_2
[conda] mkl_random 1.2.1 py38ha9443f7_2
[conda] mypy_extensions 0.4.3 py38_0
[conda] numpy 1.20.1 py38h93e21f0_0
[conda] numpy-base 1.20.1 py38h7d8b39e_0
[conda] numpydoc 1.1.0 pyhd3eb1b0_1
[conda] pytorch 1.9.1 py3.8_cuda10.2_cudnn7.6.5_0 pytorch
[conda] torchvision 0.10.1 py38_cu102 pytorch

cc @neginraoof

Start with the reproduction script in the issue, especially the torchvision Faster R-CNN MobileNet V3 export via torch.onnx.export and the ONNX Runtime session. Reproduce both the single-object reshape failure and the multi-object score mismatch using the provided images and versions. Done means the exported model runs without the reshape error and its outputs are consistent with PyTorch.