microsoft / microsoft/onnxruntime

Onnxruntime with extension has memory leak in EnableOrtCustomOps()

Open
#14,632 0 comments 0 reactions 1 assignee View on GitHub

@wenbingl is already working on this.

Since Feb 13, 2023.

core runtime
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

OS:

Ubuntu 20.04/16.04

Onnxruntime version:

1.12.1

Hi, I'm getting memory leak with ASAN build when enabling custom Ops. I haven't seen any updates in 1.13.1 release notes so I assume this leak is still there.
The code is straightforward. Any help would be appreciated on how to release the memory properly:

m_sessionOptions.SetIntraOpNumThreads(m_modelThreads);
m_sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_ALL);
m_sessionOptions.DisableCpuMemArena();

m_sessionOptions.EnableOrtCustomOps();
m_modelSession = Ort::Session(m_env, modelFile.c_str(), m_sessionOptions);

Here's the 2 leaks:

Direct leak of 56 byte(s) in 1 object(s) allocated from:
#0 0x7fbfdb814758 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:95
https://github.com/microsoft/onnxruntime-extensions/pull/1 0x7fbfcccd7d42 in OrtApis::CreateCustomOpDomain(char const*, OrtCustomOpDomain**) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x624d42)
https://github.com/microsoft/onnxruntime-extensions/pull/2 0x7fbfcf3cbb16 in RegisterCustomOps (/opt/3p/lib/libonnxruntime.so.1.12.1+0x2d18b16)
https://github.com/microsoft/onnxruntime-extensions/pull/3 0x7fbfccce24f0 in OrtApis::EnableOrtCustomOps(OrtSessionOptions*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x62f4f0)
https://github.com/microsoft/onnxruntime-extensions/pull/4 0x605318 in Ort::SessionOptions::EnableOrtCustomOps() /opt/3p/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:452
https://github.com/microsoft/onnxruntime-extensions/pull/5 0x605318 in fingerprint::TextMLFingerprintGenerator::loadModel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:88
https://github.com/microsoft/onnxruntime-extensions/pull/6 0x606d9a in fingerprint::TextMLFingerprintGenerator::TextMLFingerprintGenerator(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:64
https://github.com/microsoft/onnxruntime-extensions/pull/7 0x458605 in void std::_Construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/stl_construct.h:119
https://github.com/microsoft/onnxruntime-extensions/pull/8 0x458605 in void std::allocator_traits<std::allocator >::construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(std::allocator&, fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/alloc_traits.h:635
https://github.com/microsoft/onnxruntime-extensions/issues/9 0x458605 in std::_Sp_counted_ptr_inplace<fingerprint::TextMLFingerprintGenerator, std::allocator, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [54], int>(std::allocator, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:604
https://github.com/microsoft/onnxruntime-extensions/pull/10 0x458605 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<fingerprint::TextMLFingerprintGenerator, std::allocator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*&, std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:971
https://github.com/microsoft/onnxruntime-extensions/pull/11 0x458605 in std::__shared_ptr<fingerprint::TextMLFingerprintGenerator, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:1712
https://github.com/microsoft/onnxruntime-extensions/pull/12 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator::shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:464
https://github.com/microsoft/onnxruntime-extensions/issues/13 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator std::make_shared<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:1010
https://github.com/microsoft/onnxruntime-extensions/pull/14 0x44506d in __static_initialization_and_destruction_0 ../libs/dlp/fp20svc/test/src/DlpFp20Test.cpp:39
https://github.com/microsoft/onnxruntime-extensions/pull/15 0x6cc3dc in __libc_csu_init (/home/wxin/src/ns/dataplane/obj-asan/dlp_fp20_lib_test+0x6cc3dc)

Indirect leak of 256 byte(s) in 1 object(s) allocated from:
#0 0x7fbfdb814758 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:95
https://github.com/microsoft/onnxruntime-extensions/pull/1 0x7fbfcccfa1b2 in void std::vector<OrtCustomOp const*, std::allocator<OrtCustomOp const*> >::_M_realloc_insert<OrtCustomOp const*&>(__gnu_cxx::__normal_iterator<OrtCustomOp const**, std::vector<OrtCustomOp const*, std::allocator<OrtCustomOp const*> > >, OrtCustomOp const*&) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x6471b2)
https://github.com/microsoft/onnxruntime-extensions/pull/2 0x7fbfcccfa72c in OrtApis::CustomOpDomain_Add(OrtCustomOpDomain*, OrtCustomOp const*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x64772c)
https://github.com/microsoft/onnxruntime-extensions/pull/3 0x7fbfcf3cc48f in RegisterCustomOps (/opt/3p/lib/libonnxruntime.so.1.12.1+0x2d1948f)
https://github.com/microsoft/onnxruntime-extensions/pull/4 0x7fbfccce24f0 in OrtApis::EnableOrtCustomOps(OrtSessionOptions*) (/opt/3p/lib/libonnxruntime.so.1.12.1+0x62f4f0)
https://github.com/microsoft/onnxruntime-extensions/pull/5 0x605318 in Ort::SessionOptions::EnableOrtCustomOps() /opt/3p/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:452
https://github.com/microsoft/onnxruntime-extensions/pull/6 0x605318 in fingerprint::TextMLFingerprintGenerator::loadModel(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:88
https://github.com/microsoft/onnxruntime-extensions/pull/7 0x606d9a in fingerprint::TextMLFingerprintGenerator::TextMLFingerprintGenerator(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int) ../libs/dlp/fp20svc/src/TextMLFingerprintGenerator.cpp:64
https://github.com/microsoft/onnxruntime-extensions/pull/8 0x458605 in void std::_Construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/stl_construct.h:119
https://github.com/microsoft/onnxruntime-extensions/issues/9 0x458605 in void std::allocator_traits<std::allocator >::construct<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(std::allocator&, fingerprint::TextMLFingerprintGenerator*, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/alloc_traits.h:635
https://github.com/microsoft/onnxruntime-extensions/pull/10 0x458605 in std::_Sp_counted_ptr_inplace<fingerprint::TextMLFingerprintGenerator, std::allocator, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [54], int>(std::allocator, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:604
https://github.com/microsoft/onnxruntime-extensions/pull/11 0x458605 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<fingerprint::TextMLFingerprintGenerator, std::allocator, char const (&) [54], int>(fingerprint::TextMLFingerprintGenerator*&, std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:971
https://github.com/microsoft/onnxruntime-extensions/pull/12 0x458605 in std::__shared_ptr<fingerprint::TextMLFingerprintGenerator, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr_base.h:1712
https://github.com/microsoft/onnxruntime-extensions/issues/13 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator::shared_ptr<std::allocator, char const (&) [54], int>(std::_Sp_alloc_shared_tag<std::allocator >, char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:464
https://github.com/microsoft/onnxruntime-extensions/pull/14 0x458605 in std::shared_ptrfingerprint::TextMLFingerprintGenerator std::make_shared<fingerprint::TextMLFingerprintGenerator, char const (&) [54], int>(char const (&) [54], int&&) /opt/gcc-12/include/c++/12.2.0/bits/shared_ptr.h:1010
https://github.com/microsoft/onnxruntime-extensions/pull/15 0x44506d in __static_initialization_and_destruction_0 ../libs/dlp/fp20svc/test/src/DlpFp20Test.cpp:39
https://github.com/microsoft/onnxruntime-extensions/pull/16 0x6cc3dc in __libc_csu_init (/home/wxin/src/ns/dataplane/obj-asan/dlp_fp20_lib_test+0x6cc3dc)

SUMMARY: AddressSanitizer: 312 byte(s) leaked in 2 allocation(s).
FAIL dlp_fp20_lib_test (exit status: 1)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.