microsoft / microsoft/onnxruntime

onnxruntime on cuda fail to run 3d unet model due to 'ConvTranspose_31 Input X must be 3- or 4-dimensional. X: {1,384,16,16,16}

Open
#7,756 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
I've installed onnxruntime-gpu version 1.7.0 from pip.
I load trained model (it's unet from monai library), converted to onnx, which on cpu works just fine.
However when I execute it on GPU (RTX 3090) in a docker (based on nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04) the model fails due to error:

  File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 188, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running ConvTranspose node. Name:'ConvTranspose_31' Status Message: Input X must be 3- or 4-dimensional. X: {1,384,16,16,16}

I've seen there was a PR adding support for 3D convtranspose for gpu some time ago (https://github.com/microsoft/onnxruntime/pull/6794), which I think should be included in 1.7.0 release and I believe it should work... Can someone help me understand what is going on and is it possible to fix it somehow?

Thanks in advance for your help.

Urgency
We have to deliver inference till end of June.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 20.04
  • ONNX Runtime installed from (source or binary): pip, binary
  • ONNX Runtime version: 1.7.0
  • Python version: 3.8.5
  • Visual Studio version (if applicable):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 11.2.2 / cudnn8
  • GPU model and memory: RTX 3090 24 GB

To Reproduce
Due to NDA and data being medical records (volumetric CT scans) I cannot provide reproduction data.
However we base on monai 3D unet model and there are few CT scans available to provide some data.
Check https://docs.monai.io/en/latest/_modules/monai/networks/nets/unet.html
And probably maybe model from this could reproduce the issue
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/brats_segmentation_3d.ipynb

Expected behavior
ConvTranspose works with 5 dimension tensor on GPU (CUDA) (batch, C, H, W, D).
Currently it works only on CPU.

Screenshots
N/A

Additional context
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the referenced PR 6794 and the ConvTranspose CUDA path, then compare its behavior with the CPU path for the 5D tensor reported at ConvTranspose_31. Use the MONAI 3D UNet links or an available volumetric model to reproduce the failure; done means CUDA inference accepts the 5D ConvTranspose input as CPU inference does.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, docker, python, pytorch
Domain
ai-infra-agents, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.