microsoft / microsoft/onnxruntime

provider tensorrt crashes (Segmentation fault)

Open
#5,876 6 comments 0 reactions 1 assignee View on GitHub

@RyanUnderhill is already working on this.

Since Nov 20, 2020.

ep:TensorRT
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

**Describe the bug**
When running the application using provider tensorrt v1.5.3 , the app crashes. The app works fine with provider cuda or provider tensorrt version 1.3.

**Urgency**
none

**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian 10
- ONNX Runtime installed from (source or binary): Source
- ONNX Runtime version: 1.5.3 and master branch
- Python version:
- Visual Studio version (if applicable):
- GCC/Compiler version (if compiling from source): 8.3.0
- CUDA/cuDNN/TensorRT version: 10.2/8/7.1.3 & 11/8/7.1.3
- GPU model and memory: GXT 1080 TI, 11GB

**To Reproduce**
- Describe steps/code to reproduce the behavior.
- Attach the ONNX model to the issue (where applicable) to expedite investigation.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
Add any other context about the problem here. If the issue is about a particular model, please share the model details as well to facilitate debugging.

The app is using the C API. Here is the gdb back trace:

```
#0 0x00007ffff513ab6d in operator new(unsigned long) (n=80) at ~/workspace/onnxruntime/onnxruntime/core/providers/shared_library/provider_bridge_provider.cc:45

#1 0x00007ffff51a48fe in __gnu_cxx::new_allocator, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, true> >::allocate(unsigned long, void const*)
(this=0x7ffff530aec0 , __n=1) at /usr/include/c++/8/ext/new_allocator.h:99
#2 0x00007ffff51a48fe in std::allocator_traits, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, true> > >::allocate(std::allocator, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, true> >&, unsigned long) (__a=..., __n=1)
at /usr/include/c++/8/bits/alloc_traits.h:436
#3 0x00007ffff51a48fe in std::__detail::_Hashtable_alloc, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, true> > >::_M_allocate_node, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> > >(std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >&&) (this=0x7ffff530aec0 )
at /usr/include/c++/8/bits/hashtable_policy.h:2077
#4 0x00007ffff51a48fe in std::_Hashtable, std::allocator >, std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, std::allocator, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> > >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_emplace, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> > >(std::integral_constant, std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >&&)
(this=this@entry=0x7ffff530aec0 , __args#0=...) at /usr/include/c++/8/bits/hashtable.h:1660
#5 0x00007ffff5136ad4 in std::__detail::_Insert, std::allocator >, std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, std::allocator, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> > >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits, false>::insert, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >, void>(std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >&&)
(__v=..., this=0x7ffff530aec0 ) at /usr/include/c++/8/bits/hashtable_policy.h:1012
#6 0x00007ffff5136ad4 in std::unordered_map, std::allocator >, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)>, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> > > >::insert(std::pair, std::allocator > const, std::function > > (onnx2trt::IImporterContext*, onnx::NodeProto const&, std::vector >&)> >--Type for more, q to quit, c to continue without paging--
&&) (__x=..., this=0x7ffff530aec0 ) at /usr/include/c++/8/bits/unordered_map.h:586
#7 0x00007ffff5136ad4 in onnx2trt::(anonymous namespace)::registerBuiltinOpImporter(std::__cxx11::string, onnx2trt::NodeImporter const&) (op="Abs", importer=...)
at /home/arash/workspace/onnxruntime/cmake/external/onnx-tensorrt/builtin_op_importers.cpp:110
#8 0x00007ffff5136b76 in __static_initialization_and_destruction_0(int, int) (__priority=65535, __initialize_p=1) at /usr/include/c++/8/ext/new_allocator.h:79
#9 0x00007ffff7fe437a in call_init (l=, argc=argc@entry=1, argv=argv@entry=0x7fffffffde38, env=env@entry=0x7fffffffde48) at dl-init.c:72
#10 0x00007ffff7fe4476 in call_init (env=0x7fffffffde48, argv=0x7fffffffde38, argc=1, l=) at dl-init.c:30
#11 0x00007ffff7fe4476 in _dl_init (main_map=0x7ffff7ffe190, argc=1, argv=0x7fffffffde38, env=0x7fffffffde48) at dl-init.c:119
#12 0x00007ffff7fd60ca in _dl_start_user () at /lib64/ld-linux-x86-64.so.2
#13 0x0000000000000001 in ()
#14 0x00007fffffffe19e in ()
#15 0x0000000000000000 in ()

```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.