microsoft / microsoft/onnxruntime
GPU bug with Unpooling layer and large size inputs
@hariharans29 is already working on this.
Since Apr 8, 2020.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
Describe the bug
Large input size SegNet like model with unpooling layer (return_indices= True) fails to run on GPU.
Urgency
Using unpooling layer on GPU with large input size is blocked by this issue.
System information
- OS Platform and Distribution: Linux Ubuntu 16.04
- ONNX Runtime installed from: source
- ONNX Runtime version: 1.1.1 (2cec09a, 2020-01-22)
- Python version: 3.7
- Visual Studio version (if applicable): -
- GCC/Compiler version (if compiling from source): 5.4.0
- CUDA/cuDNN version: 10.1 / 7.6.4
- GPU model and memory: 1080 Ti / 11 Gb
To Reproduce
Link to source and models.
Compile test.cpp with ionnx class interface to onnxruntime and run it with command:
“test model_name.onnx”
Expected behavior
Should successfully run.
Additional context
All tests are carried out with C++ (CPU and GPU) and python interface (CPU). “Upsample” model converted directly from pytorch. “Unpooling” models were created in python manually (example link).
| Input size | Upscale layer | CPU, python and C++ | GPU, C++ |
|---|---|---|---|
| 128x64x1 | Unpooling | OK | OK |
| 512x256x1 | Unpooling | OK | FAIL |
| 512x256x1 | Upsample | OK | OK |
I get “Process finished with exit code 135 (interrupted by signal 7: SIGEMT)” on Ubuntu 16.04 with onnxruntime built from source.
I get “Ort::Exception at memory location 0x000000A007AFBB50” error in release mode and “Exception thrown: read access violation. Y_data was 0x111011101110111” in debug mode for similar models on Windows 10 with onnxruntime prebuild v1.1.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.