microsoft / microsoft/onnxruntime

GPU bug with Unpooling layer and large size inputs

Open
#3,113 12 comments 0 reactions 1 assignee View on GitHub

@hariharans29 is already working on this.

Since Apr 8, 2020.

Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Describe the bug
Large input size SegNet like model with unpooling layer (return_indices= True) fails to run on GPU.

Urgency
Using unpooling layer on GPU with large input size is blocked by this issue.

System information

  • OS Platform and Distribution: Linux Ubuntu 16.04
  • ONNX Runtime installed from: source
  • ONNX Runtime version: 1.1.1 (2cec09a, 2020-01-22)
  • Python version: 3.7
  • Visual Studio version (if applicable): -
  • GCC/Compiler version (if compiling from source): 5.4.0
  • CUDA/cuDNN version: 10.1 / 7.6.4
  • GPU model and memory: 1080 Ti / 11 Gb

To Reproduce
Link to source and models.
Compile test.cpp with ionnx class interface to onnxruntime and run it with command:
“test model_name.onnx”

Expected behavior
Should successfully run.

Additional context

All tests are carried out with C++ (CPU and GPU) and python interface (CPU). “Upsample” model converted directly from pytorch. “Unpooling” models were created in python manually (example link).

Input size Upscale layer CPU, python and C++ GPU, C++
128x64x1 Unpooling OK OK
512x256x1 Unpooling OK FAIL
512x256x1 Upsample OK OK

I get “Process finished with exit code 135 (interrupted by signal 7: SIGEMT)” on Ubuntu 16.04 with onnxruntime built from source.

I get “Ort::Exception at memory location 0x000000A007AFBB50” error in release mode and “Exception thrown: read access violation. Y_data was 0x111011101110111” in debug mode for similar models on Windows 10 with onnxruntime prebuild v1.1.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.