facebookresearch / facebookresearch/detectron2

About .onnx Detectron2 => executionProviders: ['wasm'] (Box: Ok, Masks: OK), executionProviders: ['webgpu'] (Box: Ok, Masks: NOK)

Open
#5,432 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

When using a model based on mask_rcnn_X_101_32x8d_FPN_3x.yaml converted to .onnx.
_(I think it was that one, please note that I am facing the same issue with good old mask_rcnn_R_50_FPN_3x.yaml)_

- I can use it as expected in a onnxruntime-web/webgpu 'wasm' provider context. (All boxes OK, All masks OK)
- I can partially use it in a onnxruntime-web/webgpu 'webgpu' provider context. (All boxes OK, **Only 1 mask OK**)

## Instructions To Reproduce the Issue:

The .onnx model was generated using commit (c69939aa85460e8135f40bce908a6cddaa73065f 21 Nov 2024)
```
"tools/deploy/export_model.py",
"--config-file", "my_conf.yml",
"--sample-image", "my_image.jpg",
"--output", "weight",
"--export-method", "tracing",
"--format", "onnx",
"MODEL.WEIGHTS", "my_model.pth",
"MODEL.DEVICE", "cuda"
```

Then I used it with success (Boxes: ALL OK, Masks: All OK) on the FE with 'wasm':
```
import * as ort from 'onnxruntime-web/webgpu';
...
const session = await ort.InferenceSession.create(MODEL_PATH, {
executionProviders: ['wasm'],

});
...
const feeds = { 'x.1': inputTensor };
const results = await session.run(feeds);
...
const boxes = results['boxes.35'].data as Float32Array; //OK
const classes = results['value.3'].data as BigInt64Array; //OK
const masks = results['value.7'].data as Float32Array; //OK (36,1,28,28)
const scores = results['value'].data as Float32Array; //OK
...
```

Then I used it with partial success (Boxes: ALL OK, Masks: **Only 1 mask OK**) on the FE with 'webgpu':
```
import * as ort from 'onnxruntime-web/webgpu';
...
const session = await ort.InferenceSession.create(MODEL_PATH, {
executionProviders: ['webgpu'], // <<<< THE ONLY CHANGE, SAME IMAGE, SAME CODE

});
...
const feeds = { 'x.1': inputTensor };
const results = await session.run(feeds);
...
const boxes = results['boxes.35'].data as Float32Array; //OK
const classes = results['value.3'].data as BigInt64Array; //OK
const masks = results['value.7'].data as Float32Array; //Shape ok (36,1,28,28) but ONLY 1 MASK WORKS
const scores = results['value'].data as Float32Array; //OK
...
```

3. __Full logs__ or other relevant observations:
**CPU Validrun logs**
```
Nohting clean log
```
Masks (reorganized output of results['value.7']) ALL GOOD:

![Image](https://github.com/user-attachments/assets/9c105705-519c-460a-bf99-3a3e5e082ae1)

**GPU problematic run logs**
```
2025-02-07 14:53:45.258599 [V:onnxruntime:Default, js_execution_provider.cc:735 JsExecutionProvider] Graph capture enable: 0
2025-02-07 14:53:45.676399 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: ReduceProd_1505
2025-02-07 14:53:45.677099 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/Gather_6
2025-02-07 14:53:45.677500 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: Gather_1596
2025-02-07 14:53:45.677799 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice
2025-02-07 14:53:45.678399 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/level_poolers.0/Gather_1
2025-02-07 14:53:45.678899 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_1
2025-02-07 14:53:45.679599 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/level_poolers.1/Gather_1
2025-02-07 14:53:45.680399 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_2
2025-02-07 14:53:45.681699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/level_poolers.2/Gather_1
2025-02-07 14:53:45.682699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_3
2025-02-07 14:53:45.683699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/level_poolers.3/Gather_1
2025-02-07 14:53:45.684699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/Gather_5
2025-02-07 14:53:45.685000 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/Gather_4
2025-02-07 14:53:45.685699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/Gather_7
2025-02-07 14:53:45.686199 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: ReduceProd_1892
2025-02-07 14:53:45.687000 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Gather
2025-02-07 14:53:45.687599 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: Gather_1993
2025-02-07 14:53:45.688000 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice
2025-02-07 14:53:45.688500 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/level_poolers.0/Gather_1
2025-02-07 14:53:45.688899 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_1
2025-02-07 14:53:45.689199 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/level_poolers.1/Gather_1
2025-02-07 14:53:45.689500 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_2
2025-02-07 14:53:45.690699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/level_poolers.2/Gather_1
2025-02-07 14:53:45.691000 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_3
2025-02-07 14:53:45.691500 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/level_poolers.3/Gather_1
2025-02-07 14:53:45.692099 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_head/Gather_3
2025-02-07 14:53:45.692599 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_head/Gather_1
2025-02-07 14:53:45.692899 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_head/Gather_2
2025-02-07 14:53:45.693299 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_head/Gather
2025-02-07 14:53:45.693799 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/box_pooler/Shape_2 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.694199 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/box_pooler/Shape_4 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.694599 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/box_pooler/Shape_6 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.695099 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/box_pooler/Shape_8 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.695399 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/box_head/flatten/Flatten because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.695899 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/mask_pooler/Shape_2 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.696299 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/mask_pooler/Shape_4 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.696799 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/mask_pooler/Shape_6 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.697099 [I:onnxruntime:Default, fallback_cpu_capability.cc:162 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: /roi_heads/mask_pooler/Shape_8 because the CPU execution path is deemed faster than overhead involved with execution on other EPs capable of executing this node
2025-02-07 14:53:45.726099 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice
2025-02-07 14:53:45.726699 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_1
2025-02-07 14:53:45.730799 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_2
2025-02-07 14:53:45.731199 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/box_pooler/Slice_3
2025-02-07 14:53:45.731599 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice
2025-02-07 14:53:45.732099 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_1
2025-02-07 14:53:45.732500 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_2
2025-02-07 14:53:45.732899 [I:onnxruntime:Default, fallback_cpu_capability.cc:86 operator()] Candidate for fallback CPU execution: /roi_heads/mask_pooler/Slice_3
2025-02-07 14:53:45.845099 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2025-02-07 14:53:45.845699 [W:onnxruntime:, session_state.cc:1170 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2025-02-07 14:53:46.198599 [V:onnxruntime:Default, conv_transpose.h:97 ConvTranspose] output_shape = {}

2025-02-07 14:53:46.199399 [V:onnxruntime:Default, conv_transpose.h:98 ConvTranspose] output_padding = {}

2025-02-07 14:53:46.237399 [V:onnxruntime:Default, js_export.cc:17 JsepOutput] JsepOutput(0, {3,1130,2260})
2025-02-07 14:53:46.239399 [V:onnxruntime:Default, js_export.cc:22 JsepOutput] JsepOutput -- data=284
2025-02-07 14:53:46.241799 [V:onnxruntime:Default, js_kernel.h:206 ComputeInternal] outputs = 1. Y.data=284.
2025-02-07 14:53:46.242899 [V:onnxruntime:Default, js_export.cc:17 JsepOutput] JsepOutput(0, {3,1130,2260})
2025-02-07 14:53:46.243599 [V:onnxruntime:Default, js_export.cc:22 JsepOutput] JsepOutput -- data=286
2025-02-07 14:53:46.245000 [V:onnxruntime:Default, js_kernel.h:206 ComputeInternal] outputs = 1. Y.data=286.
2025-02-07 14:53:46.246500 [V:onnxruntime:Default, js_export.cc:17 JsepOutput] JsepOutput(0, {3,1152,2272})
2025-02-07 14:53:46.247399 [V:onnxruntime:Default, js_export.cc:22 JsepOutput] JsepOutput -- data=288
2025-02-07 14:53:46.248500 [V:onnxruntime:Default, js_kernel.h:206 ComputeInternal] outputs = 1. Y.data=288.
2025-02-07 14:53:46.250099 [V:onnxruntime:Default, js_export.cc:17 JsepOutput] JsepOutput(0, {1,1152,2272,3})
...
```
MASKS NOK (reorganized output of results['value.7']) , only 1 mask works (same code except executionProviders)
![Image](https://github.com/user-attachments/assets/d2baf2dd-5a2e-4d32-a683-936f5ec384be)

## Expected behavior:

I expect to be able to get all masks when using the 'webgpu' execution provider

## Environment:

```
------------------------------- --------------------------------------------------------------------------------------------------------------------
PyTorch built with:
- C++ Version: 199711
- MSVC 193431937
- Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
- OpenMP 2019
- LAPACK is enabled (usually provided by MKL)
- CPU capability usage: AVX2
- CUDA Runtime 11.7
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
- CuDNN 8.5
- Magma 2.5.4
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,
```

Contributor guide

Open the contributing guide

Research direction

Start with tools/deploy/export_model.py and reproduce the exported models described in the issue. Then compare InferenceSession.create with the wasm and webgpu providers, focusing on results['value.7'] and the listed fallback logs. Done means the webgpu run produces all masks correctly, matching the wasm result for the same image and model.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python, wasm
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.