microsoft / microsoft/onnxruntime
onnxruntime on cuda fail to run 3d unet model due to 'ConvTranspose_31 Input X must be 3- or 4-dimensional. X: {1,384,16,16,16}
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
Describe the bug
I've installed onnxruntime-gpu version 1.7.0 from pip.
I load trained model (it's unet from monai library), converted to onnx, which on cpu works just fine.
However when I execute it on GPU (RTX 3090) in a docker (based on nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04) the model fails due to error:
File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 188, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running ConvTranspose node. Name:'ConvTranspose_31' Status Message: Input X must be 3- or 4-dimensional. X: {1,384,16,16,16}
I've seen there was a PR adding support for 3D convtranspose for gpu some time ago (https://github.com/microsoft/onnxruntime/pull/6794), which I think should be included in 1.7.0 release and I believe it should work... Can someone help me understand what is going on and is it possible to fix it somehow?
Thanks in advance for your help.
Urgency
We have to deliver inference till end of June.
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 20.04
- ONNX Runtime installed from (source or binary): pip, binary
- ONNX Runtime version: 1.7.0
- Python version: 3.8.5
- Visual Studio version (if applicable):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: 11.2.2 / cudnn8
- GPU model and memory: RTX 3090 24 GB
To Reproduce
Due to NDA and data being medical records (volumetric CT scans) I cannot provide reproduction data.
However we base on monai 3D unet model and there are few CT scans available to provide some data.
Check https://docs.monai.io/en/latest/_modules/monai/networks/nets/unet.html
And probably maybe model from this could reproduce the issue
https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/brats_segmentation_3d.ipynb
Expected behavior
ConvTranspose works with 5 dimension tensor on GPU (CUDA) (batch, C, H, W, D).
Currently it works only on CPU.
Screenshots
N/A
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the referenced PR 6794 and the ConvTranspose CUDA path, then compare its behavior with the CPU path for the 5D tensor reported at ConvTranspose_31. Use the MONAI 3D UNet links or an available volumetric model to reproduce the failure; done means CUDA inference accepts the 5D ConvTranspose input as CPU inference does.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, docker, python, pytorch
- Domain
- ai-infra-agents, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100