OpenVINO EP inferencing issue for some combinations of model/platforms/device

Nobody has claimed this yet.

I’m currently testing the following models using OpenVINO Execution Provider:
• Tiny YoloV2 from ONNX Model Zoo https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/tiny-yolov2
• Tiny YoloV3 from ONNX Model Zoo https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/tiny-yolov3
• Faster RCNN from ONNX Model Zoo https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/faster-rcnn
• Human Pose Estimation from OpenVINO Model Zoo (Public Models) https://github.com/opencv/open_model_zoo/blob/master/models/public/human-pose-estimation-3d-0001/description/human-pose-estimation-3d-0001.md
On this different platform:
• Windows 10 Laptop with i7-8650U
• IEI Tank Ubuntu 16 with i7-6700 and mustang card (VAD-M_FP16)
• Azure NC6 Ubuntu 18 with Xeon E5-2690
And I found the below issue on some configuration and I would like to check with you if they’re expected or not.

Additionally, I notice some issue on Python closing phase (DLL Detach) on windows where it crashes mostly systematically for GPU inferencing (sometimes it’s enough to load the model, other time you need to inference also something) and on Linux where VAD-M_FP16 hangs on exit (CTRL+C is needed).

Model inferencing issues:
- **Human Pose Estimation.** Works on all configuration on Windows Laptop, _fails on all configuration on both Linux deployments_ (IEI Tank and Azure) with the same error (see full logs attached):

```
File "/usr/local/lib/python3.6/dist-packages/onnxruntime/capi/session.py", line 177, in _load_model
self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /data/onnx/onnxruntime-1.3.0/onnxruntime/core/providers/openvino/backend_utils.cc:69 std::shared_ptr onnxruntime::openvino_ep::backend_utils::CreateCNNNetwork(const onnx::ModelProto&, std::__cxx11::string, InferenceEngine::Precision) [OpenVINO-EP] Exception thrown while making IE::CNNNetwork: All operations in nGraph function should have unique friendly names!
```

- **Tiny YoloV3** could be scored only on CPU_FP32. _All other configuration GPU_FP16, GPU_FP32, MYRIAD_FP16, and VAD-M_FP16 return errors_ (see full logs attached).

- **GPU_FP16 (Win Laptop and IEI Tank)**
```
File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_4 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_3' Status Message: Check 'element::Type::merge(element_type, element_type, get_input_element_type(i))' failed at /data/onnx/onnxruntime-1.3.0/build/Linux/Release/ngraph/src/project_ngraph/src/ngraph/node.cpp:889:
While validating node 'v1::Add Add_8653(Parameter_8649[0]:f16{1,26,26,3,2}, Convert_8652[0]:f32{26,26,1,2}) -> (f32{1,26,26,3,2})':
Argument element types are inconsistent.
```

- **GPU_FP32 (Win Laptop and IEI Tank)**

```
File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_4 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_3' Status Message: /data/onnx/onnxruntime-1.3.0/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_3Error has occured for: eltwise:TFNodes/yolo_evaluation_layer_1/truediv_8:0
Sizes equal or broadcast is possible(true) should be false
Invalid input shapes
```

- **MYRIAD_FP16 (Win Laptop)**

```
File "C:\Python\Python37_64\lib\site-packages\onnxruntime\capi\session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_7 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_7_6' Status Message: D:\onnx\onnxruntime-1.3.0\onnxruntime\core\providers\openvino\backends\basic_backend.cc:41 __cdecl onnxruntime::openvino_ep::BasicBackend::BasicBackend(const class onnx::ModelProto &,struct onnxruntime::openvino_ep::GlobalContext &,const struct onnxruntime::openvino_ep::SubGraphContext &) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_7_6Input Parameter_12898 has unexpected layout C
```
- **VAD-M_FP16 (IEI Tank))**
```
File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_26 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_26_25' Status Message: /data/onnx/onnxruntime-1.3.0/onnxruntime/core/providers/openvino/backends/vadm_backend.cc:64 onnxruntime::openvino_ep::VADMBackend::VADMBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while creating InferRequest object: AssertionFailed: hddlBlob->getSize() >= offset + size
```

- **Faster RCNN** could be scored only on CPU_FP32. _All other configuration GPU_FP16, GPU_FP32, MYRIAD_FP16, and VAD-M_FP16 return errors (see full logs attached).

- **GPU_FP32 & GPU_FP16** (Win Laptop and IEI Tank)
```
File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 177, in _load_model
self._sess.load_model(providers)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /data/onnx/onnxruntime-1.3.0/onnxruntime/core/providers/openvino/backends/basic_backend.cc:41 onnxruntime::openvino_ep::BasicBackend::BasicBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_41_40Error has occured for: concat:415
Input size dim: 1(=2) is not equal to: input memory dim: 1(=1)
Every input must have the same size
```
- **MYRIAD_FP16 (Win Laptop)** (maybe is a too big model for the myriad)
```
File "C:\Python\Python37_64\lib\site-packages\onnxruntime\capi\session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_8 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_8_7' Status Message: D:\onnx\onnxruntime-1.3.0\onnxruntime\core\providers\openvino\backends\basic_backend.cc:41 __cdecl onnxruntime::openvino_ep::BasicBackend::BasicBackend(const class onnx::ModelProto &,struct onnxruntime::openvino_ep::GlobalContext &,const struct onnxruntime::openvino_ep::SubGraphContext &) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_8_7Failed to create input FIFO: NC_OUT_OF_MEMORY
```
- **VAD-M_FP16 (IEI Tank)**
```
File "/usr/local/lib/python3.5/dist-packages/onnxruntime/capi/session.py", line 111, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running OpenVINO-EP-subgraph_17 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_17_16' Status Message: /data/onnx/onnxruntime-1.3.0/onnxruntime/core/providers/openvino/backends/vadm_backend.cc:52 onnxruntime::openvino_ep::VADMBackend::VADMBackend(const onnx::ModelProto&, onnxruntime::openvino_ep::GlobalContext&, const onnxruntime::openvino_ep::SubGraphContext&) [OpenVINO-EP] Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_17_16Input Parameter_1539 has unexpected layout CHW
```
[ErrorReport.zip](https://github.com/microsoft/onnxruntime/files/4794213/ErrorReport.zip)

Start by reviewing the OpenVINO provider paths named in the traces, including core/providers/openvino/backend_utils.cc, basic_backend.cc, and vadm_backend.cc, then reproduce the model/platform matrix using the attached ErrorReport.zip. Compare the reported failures across CPU, GPU, MYRIAD, and VAD-M configurations; done would require a confirmed diagnosis and agreed behavior or fixes for the affected combinations.

Written by the indexing model from the issue text.