microsoft / microsoft/onnxruntime

onnxruntime_provider_test test failure on 1.26.0

Open
#28,664 2 comments 1 reaction 2 assignees View on GitHub

@tianleiwu is already working on this.

Since Jul 31, 2026.

  • #31346 by @copilot-swe-agent — open
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

This is a forward of a downstream report already reported on [Gentoo bugzilla](https://bugs.gentoo.org/975584)

The maintainer is not able to reproduce the issue.
Already attached on the Gentoo Bugzilla both `build.log` and `LastTest.log`

Additionally, I tried to recompile with asan and by doing `onnxruntime_provider_test` I can see:

```
=================================================================
==2779807==ERROR: AddressSanitizer: container-overflow on address 0x7c2bb6c9ccb8 at pc 0x7ffbb892680d bp 0x7fff0bcfc2c0 sp 0x7fff0bcfba68
WRITE of size 24 at 0x7c2bb6c9ccb8 thread T0
#0 0x7ffbb892680c in memcpy (/usr/lib/gcc/x86_64-pc-linux-gnu/16/libasan.so.8+0x12680c) (BuildId: 63bc9dc6067b6ccf3e6e0399ffe0119d42bd745a)
#1 0x7ffbb853cc29 in onnx::AttributeProto::Impl_::Impl_(google::protobuf::internal::InternalVisibility, google::protobuf::Arena*, onnx::AttributeProto::Impl_ const&, onnx::AttributeProto const&) (/usr/lib64/libonnx.so+0x33cc29) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#2 0x7ffbb8537f29 in onnx::AttributeProto::AttributeProto(google::protobuf::Arena*, onnx::AttributeProto const&) (/usr/lib64/libonnx.so+0x337f29) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#3 0x5646285d414c in onnx::AttributeProto::AttributeProto(onnx::AttributeProto const&) /usr/include/onnx/onnx-ml.pb.h:5362
#4 0x5646285d4836 in onnx::OpSchema::Attribute::Attribute(onnx::OpSchema::Attribute const&) /usr/include/onnx/defs/schema.h:432
#5 0x564628de1fec in std::pair, std::allocator > const, onnx::OpSchema::Attribute>::pair(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/stl_pair.h:315
#6 0x564628de2068 in _ZSt12construct_atISt4pairIKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEN4onnx8OpSchema9AttributeEEJRKSB_EQaant20is_unbounded_array_vIT_ErqXgsnwcvPvLi0E_SE_pispcl7declvalIT0_EEEEEPSE_SH_DpOSG_ /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/stl_construct.h:110
#7 0x564628de08d8 in void std::allocator_traits, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::construct, std::allocator > const, onnx::OpSchema::Attribute>, std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&>(std::allocator, std::allocator > const, onnx::OpSchema::Attribute>, true> >&, std::pair, std::allocator > const, onnx::OpSchema::Attribute>*, std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/alloc_traits.h:716
#8 0x564628de08d8 in std::__detail::_Hash_node, std::allocator > const, onnx::OpSchema::Attribute>, true>* std::__detail::_Hashtable_alloc, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::_M_allocate_node, std::allocator > const, onnx::OpSchema::Attribute> const&>(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable_policy.h:1575
#9 0x564628dde414 in std::__detail::_Hash_node, std::allocator > const, onnx::OpSchema::Attribute>, true>* std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > >::operator(), std::allocator > const, onnx::OpSchema::Attribute> const&>(std::pair, std::allocator > const, onnx::OpSchema::Attribute> const&) const /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable_policy.h:232
#10 0x564628ddb369 in void std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_assign, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&, std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > > >(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&, std::__detail::_AllocNode, std::allocator > const, onnx::OpSchema::Attribute>, true> > >&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:1674
#11 0x564628dd6d39 in void std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_assign, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&>(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:453
#12 0x564628dd337b in std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_Hashtable(std::_Hashtable, std::allocator >, std::pair, std::allocator > const, onnx::OpSchema::Attribute>, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> >, std::__detail::_Select1st, std::equal_to, std::allocator > >, std::hash, std::allocator > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/hashtable.h:1782
#13 0x564628dcf994 in std::unordered_map, std::allocator >, onnx::OpSchema::Attribute, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> > >::unordered_map(std::unordered_map, std::allocator >, onnx::OpSchema::Attribute, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, onnx::OpSchema::Attribute> > > const&) /usr/lib/gcc/x86_64-pc-linux-gnu/16/include/g++-v16/bits/unordered_map.h:191
#14 0x564628dcfaa9 in onnx::OpSchema::OpSchema(onnx::OpSchema const&) /usr/include/onnx/defs/schema.h:147
#15 0x7ffbb83cecf8 in onnx::OpSchema onnx::GetOpSchema() (/usr/lib64/libonnx.so+0x1cecf8) (BuildId: c1337901977b519aa741436b1d63758b3105f363)
#16 0x56462a411deb in onnx::OpSet_Onnx_ver9::ForEachSchema(std::function const&) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2617deb) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#17 0x56462a41bcb6 in onnx::RegisterOnnxOperatorSetSchema() (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2621cb6) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#18 0x56462a41ce2b in onnxruntime::Environment::Initialize(std::unique_ptr >, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*)::{lambda()#1}::operator()() const [clone .isra.0] (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2622e2b) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#19 0x7ffbb7b8cad5 (/usr/lib64/libc.so.6+0x9bad5) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#20 0x7ffbb7b8cb58 in pthread_once (/usr/lib64/libc.so.6+0x9bb58) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#21 0x56462a41f509 in onnxruntime::Environment::Initialize(std::unique_ptr >, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2625509) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#22 0x56462a421199 in onnxruntime::Environment::Create(std::unique_ptr >, std::unique_ptr >&, OrtThreadingOptions const*, bool, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2627199) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#23 0x56462a3910f1 in OrtEnv::GetOrCreateInstance(OrtEnv::LoggingManagerConstructionInfo const&, onnxruntime::common::Status&, OrtThreadingOptions const*, OrtKeyValuePairs const*) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x25970f1) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#24 0x56462a373a07 in OrtApis::CreateEnvWithGlobalThreadPools(OrtLoggingLevel, char const*, OrtThreadingOptions const*, OrtEnv**) (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x2579a07) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)
#25 0x56462a2bc849 in Ort::Env::Env(OrtThreadingOptions const*, OrtLoggingLevel, char const*) /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/include/onnxruntime/core/session/onnxruntime_cxx_inline.h:938
#26 0x56462a2bdedf in ortenv_setup /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/onnxruntime/test/unittest_main/test_main.cc:82
#27 0x56462a2bfc8d in main /var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/onnxruntime/test/unittest_main/test_main.cc:196
#28 0x7ffbb7b183bd (/usr/lib64/libc.so.6+0x273bd) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#29 0x7ffbb7b184da in __libc_start_main (/usr/lib64/libc.so.6+0x274da) (BuildId: a096e856a7ce50b511403b1be4184b01a11dfead)
#30 0x56462807e234 in _start (/var/tmp/portage/sci-libs/onnxruntime-1.26.0-r1/work/onnxruntime-1.26.0/cmake_build/onnxruntime_provider_test+0x284234) (BuildId: 62a8dbd80c7737ab42d96cc6bfd4171777a80c24)

0x7c2bb6c9ccb8 is located 8 bytes inside of 32-byte region [0x7c2bb6c9ccb0,0x7c2bb6c9ccd0)
allocated by thread T0 here:
#0 0x7ffbb892a25f in operator new(unsigned long) (/usr/lib/gcc/x86_64-pc-linux-gnu/16/libasan.so.8+0x12a25f) (BuildId: 63bc9dc6067b6ccf3e6e0399ffe0119d42bd745a)
#1 0x5646288e0ce6 in google::protobuf::internal::AllocateAtLeast(unsigned long) /usr/include/google/protobuf/port.h:148
#2 0x5646288ffbd5 in google::protobuf::RepeatedField::GrowNoAnnotate(bool, int, int) /usr/include/google/protobuf/repeated_field.h:1250
#3 0x5646288fba13 in google::protobuf::RepeatedField::Grow(bool, int, int) /usr/include/google/protobuf/repeated_field.h:1289
#4 0x7ffbb853c980 in google::protobuf::RepeatedField::RepeatedField(google::protobuf::Arena*, google::protobuf::RepeatedField const&) (/usr/lib64/libonnx.so+0x33c980) (BuildId: c1337901977b519aa741436b1d63758b3105f363)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (/usr/lib64/libonnx.so+0x33cc29) (BuildId: c1337901977b519aa741436b1d63758b3105f363) in onnx::AttributeProto::Impl_::Impl_(google::protobuf::internal::InternalVisibility, google::protobuf::Arena*, onnx::AttributeProto::Impl_ const&, onnx::AttributeProto const&)
Shadow bytes around the buggy address:
0x7c2bb6c9ca00: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00
0x7c2bb6c9ca80: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa
0x7c2bb6c9cb00: 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00 00 00
0x7c2bb6c9cb80: fa fa 00 00 00 00 fa fa 00 00 00 00 fa fa 00 00
0x7c2bb6c9cc00: 00 00 fa fa 00 00 00 00 fa fa 00 00 00 02 fa fa
=>0x7c2bb6c9cc80: 00 00 00 02 fa fa 00[fc]fc fc fa fa fa fa fa fa
0x7c2bb6c9cd00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9cd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9ce00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9ce80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x7c2bb6c9cf00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==2779807==ABORTING
```

I did not analyze the output and I don't know if the overflow occurs in the tests itself or it is in the library and then it is a security issue.

Since I'm the only one that can reproduce the issue if I can help further, please let me know.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.