microsoft / microsoft/onnxruntime
onnxruntime_provider_test test failure on 1.26.0
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
This is a forward of a downstream report already reported on [Gentoo bugzilla](https://bugs.gentoo.org/975584)
The maintainer is not able to reproduce the issue.
Already attached on the Gentoo Bugzilla both `build.log` and `LastTest.log`
Additionally, I tried to recompile with asan and by doing `onnxruntime_provider_test` I can see:
```
=================================================================
==2779807==ERROR: AddressSanitizer: container-overflow on address 0x7c2bb6c9ccb8 at pc 0x7ffbb892680d bp 0x7fff0bcfc2c0 sp 0x7fff0bcfba68
WRITE of size 24 at 0x7c2bb6c9ccb8 thread T0
#0 0x7ffbb892680c in memcpy (/usr/lib/gcc/x86_64-pc-linux-gnu/16/libasan.so.8+0x12680c) (BuildId: 63bc9dc6067b6ccf3e6e0399ffe0119d42bd745a)
#1 0x7ffbb853cc29 in onnx::AttributeProto::Impl_::Impl_(google::protobuf::internal::InternalVisibility, google::protobuf::Arena*, onnx::AttributeProto::Impl_ const&, onnx::AttributeProto const&) (/usr/lib64/libonnx.so+0x33cc29) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#2 0x7ffbb8537f29 in onnx::AttributeProto::AttributeProto(google::protobuf::Arena*, onnx::AttributeProto const&) (/usr/lib64/libonnx.so+0x337f29) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#3 0x5646285d414c in onnx::AttributeProto::AttributeProto(onnx::AttributeProto const&) /usr/include/onnx/onnx-ml.pb.h:5362
#4 0x5646285d4836 in onnx::OpSchema::Attribute::Attribute(onnx::OpSchema::Attribute const&) /usr/include/onnx/defs/schema.h:432
#5 0x564628de1fec in std::pair, std::allocator > const, onnx::OpSchema::Attribute>::pair(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/stl_pair.h:315
#6 0x564628de2068 in _ZSt12construct_atISt4pairIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN4onnx8OpSchema9AttributeEEJRKSB_EQaant20is_unbounded_array_vIT_ErqXgsnwcvPvLi0E_SE_pispcl7declvalIT0_EEEEEPSE_SH_DpOSG_ /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/stl_construct.h:110
#7 0x564628de08d8 in void std::allocator_traits, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::construct, std::allocator > const, onnx::OpSchema::Attribute>, std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&>(std::allocator, std::allocator > const, onnx::OpSchema::Attribute>, true> >&, std::pair, std::allocator > const, onnx::OpSchema::Attribute>*, std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/alloc_traits.h:716
#8 0x564628de08d8 in std::__detail::_Hash_node, std::allocator > const, onnx::OpSchema::Attribute>, true>* std::__detail::_Hashtable_alloc, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::_M_allocate_node, std::allocator > const, onnx::OpSchema::Attribute> const&>(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable_policy.h:1575
#9 0x564628dde414 in std::__detail::_Hash_node, std::allocator > const, onnx::OpSchema::Attribute>, true>* std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::operator(), std::allocator > const, onnx::OpSchema::Attribute> const&>(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) const /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable_policy.h:232
#10 0x564628ddb369 in void std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_assign, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&, std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > > >(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&, std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > >&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:1674
#11 0x564628dd6d39 in void std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_assign, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&>(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:453
#12 0x564628dd337b in std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_Hashtable(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:1782
#13 0x564628dcf994 in std::unordered_map, std::allocator >, onnx::OpSchema::Attribute, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> > >::unordered_map(std::unordered_map, std::allocator >, onnx::OpSchema::Attribute, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> > > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/unordered_map.h:191
#14 0x564628dcfaa9 in onnx::OpSchema::OpSchema(onnx::OpSchema const&) /usr/include/onnx/defs/schema.h:147
#15 0x7ffbb83cecf8 in onnx::OpSchema onnx::GetOpSchema() (/usr/lib64/libonnx.so+0x1cecf8) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#16 0x56462a411deb in onnx::OpSet_Onnx_ver9::ForEachSchema(std::function const&) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2617deb) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#17 0x56462a41bcb6 in onnx::RegisterOnnxOperatorSetSchema() (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2621cb6) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#18 0x56462a41ce2b in onnxruntime::Environment::Initialize(std::unique_ptr >, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*)::{lambda()#1}::operator()() const [clone .isra.0] (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2622e2b) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#19 0x7ffbb7b8cad5 (/usr/lib64/libc.so.6+0x9bad5) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#20 0x7ffbb7b8cb58 in pthread_once (/usr/lib64/libc.so.6+0x9bb58) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#21 0x56462a41f509 in onnxruntime::Environment::Initialize(std::unique_ptr >, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2625509) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#22 0x56462a421199 in onnxruntime::Environment::Create(std::unique_ptr >, std::unique_ptr >&, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2627199) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#23 0x56462a3910f1 in OrtEnv::GetOrCreateInstance(OrtEnv::LoggingManagerConstructionInfo const&, onnxruntime::common::Status&, OrtThreadingOptions const*, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x25970f1) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#24 0x56462a373a07 in OrtApis::CreateEnvWithGlobalThreadPools(OrtLoggingLevel, char const*, OrtThreadingOptions const*, OrtEnv**) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2579a07) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#25 0x56462a2bc849 in Ort::Env::Env(OrtThreadingOptions const*, OrtLoggingLevel, char const*) /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:938
#26 0x56462a2bdedf in ortenv_setup /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/onnxruntime/test/unittest_main/test_main.cc:82
#27 0x56462a2bfc8d in main /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/onnxruntime/test/unittest_main/test_main.cc:196
#28 0x7ffbb7b183bd (/usr/lib64/libc.so.6+0x273bd) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#29 0x7ffbb7b184da in __libc_start_main (/usr/lib64/libc.so.6+0x274da) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#30 0x56462807e234 in _start (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x284234) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
0x7c2bb6c9ccb8 is located 8 bytes inside of 32-byte region [0x7c2bb6c9ccb0,0x7c2bb6c9ccd0)
allocated by thread T0 here:
#0 0x7ffbb892a25f in operator new(unsigned long) (/usr/lib/gcc/x86_64-pc-linux-gnu/16/libasan.so.8+0x12a25f) (BuildId: 63bc9dc6067b6ccf3e6e0399ffe0119d42bd745a)
#1 0x5646288e0ce6 in google::protobuf::internal::AllocateAtLeast(unsigned long) /usr/include/google/protobuf/port.h:148
#2 0x5646288ffbd5 in google::protobuf::RepeatedField::GrowNoAnnotate(bool, int, int) /usr/include/google/protobuf/repeated_field.h:1250
#3 0x5646288fba13 in google::protobuf::RepeatedField::Grow(bool, int, int) /usr/include/google/protobuf/repeated_field.h:1289
#4 0x7ffbb853c980 in google::protobuf::RepeatedField::RepeatedField(google::protobuf::Arena*, google::protobuf::RepeatedField const&) (/usr/lib64/libonnx.so+0x33c980) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (/usr/lib64/libonnx.so+0x33cc29) (BuildId: c1337901977b519aa741436b1d63758b3105f363) in onnx::AttributeProto::Impl_::Impl_(google::protobuf::internal::InternalVisibility, google::protobuf::Arena*, onnx::AttributeProto::Impl_ const&, onnx::AttributeProto const&)
Shadow bytes around the buggy address:
0x7c2bb6c9ca00: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00
0x7c2bb6c9ca80: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa
0x7c2bb6c9cb00: 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00
0x7c2bb6c9cb80: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00
0x7c2bb6c9cc00: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 02 fa fa
=>0x7c2bb6c9cc80: 00 00 00 02 fa fa 00[fc]fc fc fa fa fa fa fa fa
0x7c2bb6c9cd00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9cd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9ce00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9ce80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9cf00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2779807==ABORTING
```
I did not analyze the output and I don't know if the overflow occurs in the tests itself or it is in the library and then it is a security issue.
Since I'm the only one that can reproduce the issue if I can help further, please let me know.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.