microsoft / microsoft/onnxruntime
Onnxruntime with extension has memory leak in EnableOrtCustomOps()
@wenbingl is already working on this.
Since Feb 13, 2023.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
OS:
Ubuntu 20.04/16.04
Onnxruntime version:
1.12.1
Hi, I'm getting memory leak with ASAN build when enabling custom Ops. I haven't seen any updates in 1.13.1 release notes so I assume this leak is still there.
The code is straightforward. Any help would be appreciated on how to release the memory properly:
m_sessionOptions.SetIntraOpNumThreads(m_modelThreads);
m_sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
m_sessionOptions.DisableCpuMemArena();
m_sessionOptions.EnableOrtCustomOps();
m_modelSession = Ort::Session(m_env, modelFile.c_str(), m_sessionOptions);
Here's the 2 leaks:
Direct leak of 56 byte(s) in 1 object(s) allocated from:
#0 0x7fbfdb814758 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:95
https://github.com/microsoft/onnxruntime-extensions/pull/1 0x7fbfcccd7d42 in OrtApis::CreateCustomOpDomain(char const*, OrtCustomOpDomain**) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x624d42)
https://github.com/microsoft/onnxruntime-extensions/pull/2 0x7fbfcf3cbb16 in RegisterCustomOps (/opt/3p/lib/libonnxruntime.so.1.12.1+0x2d18b16)
https://github.com/microsoft/onnxruntime-extensions/pull/3 0x7fbfccce24f0 in OrtApis::EnableOrtCustomOps(OrtSessionOptions*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x62f4f0)
https://github.com/microsoft/onnxruntime-extensions/pull/4 0x605318 in Ort::SessionOptions::EnableOrtCustomOps() /opt/3p/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:452
https://github.com/microsoft/onnxruntime-extensions/pull/5 0x605318 in fingerprint::TextMLFingerprintGenerator::loadModel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:88
https://github.com/microsoft/onnxruntime-extensions/pull/6 0x606d9a in fingerprint::TextMLFingerprintGenerator::TextMLFingerprintGenerator(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:64
https://github.com/microsoft/onnxruntime-extensions/pull/7 0x458605 in void std::_Construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/stl_construct.h:119
https://github.com/microsoft/onnxruntime-extensions/pull/8 0x458605 in void std::allocator_traits<std::allocator >::construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(std::allocator&, fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/alloc_traits.h:635
https://github.com/microsoft/onnxruntime-extensions/issues/9 0x458605 in std::_Sp_counted_ptr_inplace<fingerprint::TextMLFingerprintGenerator, std::allocator, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [54], int>(std::allocator, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:604
https://github.com/microsoft/onnxruntime-extensions/pull/10 0x458605 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<fingerprint::TextMLFingerprintGenerator, std::allocator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*&, std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:971
https://github.com/microsoft/onnxruntime-extensions/pull/11 0x458605 in std::__shared_ptr<fingerprint::TextMLFingerprintGenerator, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:1712
https://github.com/microsoft/onnxruntime-extensions/pull/12 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator::shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:464
https://github.com/microsoft/onnxruntime-extensions/issues/13 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator std::make_shared<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:1010
https://github.com/microsoft/onnxruntime-extensions/pull/14 0x44506d in __static_initialization_and_destruction_0 ../libs/dlp/fp20svc/test/src/DlpFp20Test.cpp:39
https://github.com/microsoft/onnxruntime-extensions/pull/15 0x6cc3dc in __libc_csu_init (/home/wxin/src/ns/dataplane/obj-asan/dlp_fp20_lib_test+0x6cc3dc)
Indirect leak of 256 byte(s) in 1 object(s) allocated from:
#0 0x7fbfdb814758 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:95
https://github.com/microsoft/onnxruntime-extensions/pull/1 0x7fbfcccfa1b2 in void std::vector<OrtCustomOp const*, std::allocator<OrtCustomOp const*> >::_M_realloc_insert<OrtCustomOp const*&>(__gnu_cxx::__normal_iterator<OrtCustomOp const**, std::vector<OrtCustomOp const*, std::allocator<OrtCustomOp const*> > >, OrtCustomOp const*&) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x6471b2)
https://github.com/microsoft/onnxruntime-extensions/pull/2 0x7fbfcccfa72c in OrtApis::CustomOpDomain_Add(OrtCustomOpDomain*, OrtCustomOp const*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x64772c)
https://github.com/microsoft/onnxruntime-extensions/pull/3 0x7fbfcf3cc48f in RegisterCustomOps (/opt/3p/lib/libonnxruntime.so.1.12.1+0x2d1948f)
https://github.com/microsoft/onnxruntime-extensions/pull/4 0x7fbfccce24f0 in OrtApis::EnableOrtCustomOps(OrtSessionOptions*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x62f4f0)
https://github.com/microsoft/onnxruntime-extensions/pull/5 0x605318 in Ort::SessionOptions::EnableOrtCustomOps() /opt/3p/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:452
https://github.com/microsoft/onnxruntime-extensions/pull/6 0x605318 in fingerprint::TextMLFingerprintGenerator::loadModel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:88
https://github.com/microsoft/onnxruntime-extensions/pull/7 0x606d9a in fingerprint::TextMLFingerprintGenerator::TextMLFingerprintGenerator(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:64
https://github.com/microsoft/onnxruntime-extensions/pull/8 0x458605 in void std::_Construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/stl_construct.h:119
https://github.com/microsoft/onnxruntime-extensions/issues/9 0x458605 in void std::allocator_traits<std::allocator >::construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(std::allocator&, fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/alloc_traits.h:635
https://github.com/microsoft/onnxruntime-extensions/pull/10 0x458605 in std::_Sp_counted_ptr_inplace<fingerprint::TextMLFingerprintGenerator, std::allocator, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [54], int>(std::allocator, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:604
https://github.com/microsoft/onnxruntime-extensions/pull/11 0x458605 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<fingerprint::TextMLFingerprintGenerator, std::allocator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*&, std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:971
https://github.com/microsoft/onnxruntime-extensions/pull/12 0x458605 in std::__shared_ptr<fingerprint::TextMLFingerprintGenerator, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:1712
https://github.com/microsoft/onnxruntime-extensions/issues/13 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator::shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:464
https://github.com/microsoft/onnxruntime-extensions/pull/14 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator std::make_shared<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:1010
https://github.com/microsoft/onnxruntime-extensions/pull/15 0x44506d in __static_initialization_and_destruction_0 ../libs/dlp/fp20svc/test/src/DlpFp20Test.cpp:39
https://github.com/microsoft/onnxruntime-extensions/pull/16 0x6cc3dc in __libc_csu_init (/home/wxin/src/ns/dataplane/obj-asan/dlp_fp20_lib_test+0x6cc3dc)
SUMMARY: AddressSanitizer: 312 byte(s) leaked in 2 allocation(s).
FAIL dlp_fp20_lib_test (exit status: 1)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.