apache / apache/arrow

[C++] Arrow test 'arrow-utility-test' contains container-overflow error

Open
#50,244 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When I verify arrow test after https://github.com/apache/arrow/issues/48094 resolved, the test 'arrow-utility-test' failed with container-overflow error again, could you please take a look?

Steps to reproduce:
```
1. Open VS2022 x86 Native Command Prompt tools
2. Clone repo and checkout 0249a2e
3. set _CL_=/fsanitize=address /GS- /wd5072 & set _LINK_=/InferASanLibs /incremental:no /debug
4. cd to build folder: cd /d C:\gitP\apache\arrow\cpp\build_amd64
5. cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.26100.0 -DARROW_BUILD_TESTS=ON -DARROW_ACERO=ON -DARROW_FILESYSTEM=ON -DARROW_PARQUET=OFF -DARROW_SUBSTRAIT=ON -DARROW_DATASET=ON
6. msbuild /m /p:Platform=x64 /p:Configuration=Release arrow.sln /t:Rebuild
7.set ARROW_TEST_DATA=C:\gitP\apache\arrow\testing\data
8.ctest -C Release --output-on-failure -R "arrow-utility-test"
```

Memory safety issue reported by Address Sanitizer:
```
==6460==ERROR: AddressSanitizer: container-overflow on address 0x1298940cf031 at pc 0x7ffed5c28155 bp 0x00fb34d6e0c0 sp 0x00fb34d6d848
READ of size 42 at 0x1298940cf031 thread T0
#0 0x7ffed5c28154 in MemcmpInterceptorCommon(void *, int (__cdecl *)(void const *, void const *, unsigned __int64), void const *, void const *, unsigned __int64) C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:896
#1 0x7ffed5c20600 in memcmp C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:928
#2 0x7ff72ef855c8 in testing::internal::CmpHelperEQ>, class std::basic_string_view>>(char const *, char const *, class std::basic_string_view> const &, class std::basic_string_view> const &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1405a55c8)
#3 0x7ff72ef88189 in arrow::util::test::IsSecurelyCleared(class std::basic_string_view> const &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1405a8189)
#4 0x7ff72ef72ffd in arrow::util::test::TestSecureString_AssertSecurelyCleared_Test::TestBody(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140592ffd)
#5 0x7fff05898986 in testing::internal::HandleSehExceptionsInMethodIfSupported(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180008986)
#6 0x7fff058984db in testing::internal::HandleExceptionsInMethodIfSupported(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800084db)
#7 0x7fff0590645c in testing::Test::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x18007645c)
#8 0x7fff05906721 in testing::TestInfo::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076721)
#9 0x7fff05906b43 in testing::TestSuite::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076b43)
#10 0x7fff05908491 in testing::internal::UnitTestImpl::RunAllTests(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180078491)
#11 0x7fff058989e6 in testing::internal::HandleSehExceptionsInMethodIfSupported(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800089e6)
#12 0x7fff058988d9 in testing::internal::HandleExceptionsInMethodIfSupported(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800088d9)
#13 0x7fff05906fab in testing::UnitTest::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076fab)
#14 0x7ff72eef1dce in main (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140511dce)
#15 0x7ff72f23ccae in invoke_main C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
#16 0x7ff72f23ccae in __scrt_common_main_seh C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#17 0x7fff8bf5e956 (C:\WINDOWS\System32\KERNEL32.DLL+0x18002e956)
#18 0x7fff8db47c1b (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180087c1b)

0x1298940cf040 is located 0 bytes after 48-byte region [0x1298940cf010,0x1298940cf040)
allocated by thread T0 here:
#0 0x7ff72f23baf5 in operator new(unsigned __int64) C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win_new_scalar_thunk.cpp:40
#1 0x7ff72ea65951 in std::_Allocate<16, struct std::_Default_allocate_traits>(unsigned __int64) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140085951)
#2 0x7ff72ea659e9 in std::basic_string, class std::allocator>::_Allocate_for_capacity<0>(class std::allocator &, unsigned __int64 &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1400859e9)
#3 0x7ff72ea65f96 in std::basic_string, class std::allocator>::_Construct<2, char const *>(char const *const, unsigned __int64) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140085f96)
#4 0x7ff72ef72f43 in arrow::util::test::TestSecureString_AssertSecurelyCleared_Test::TestBody(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140592f43)
#5 0x7fff05898986 in testing::internal::HandleSehExceptionsInMethodIfSupported(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180008986)
#6 0x7fff058984db in testing::internal::HandleExceptionsInMethodIfSupported(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800084db)
#7 0x7fff0590645c in testing::Test::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x18007645c)
#8 0x7fff05906721 in testing::TestInfo::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076721)
#9 0x7fff05906b43 in testing::TestSuite::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076b43)
#10 0x7fff05908491 in testing::internal::UnitTestImpl::RunAllTests(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180078491)
#11 0x7fff058989e6 in testing::internal::HandleSehExceptionsInMethodIfSupported(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800089e6)
#12 0x7fff058988d9 in testing::internal::HandleExceptionsInMethodIfSupported(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800088d9)
#13 0x7fff05906fab in testing::UnitTest::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076fab)
#14 0x7ff72eef1dce in main (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140511dce)
#15 0x7ff72f23ccae in invoke_main C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
#16 0x7ff72f23ccae in __scrt_common_main_seh C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
#17 0x7fff8bf5e956 (C:\WINDOWS\System32\KERNEL32.DLL+0x18002e956)
#18 0x7fff8db47c1b (C:\WINDOWS\SYSTEM32\ntdll.dll+0x180087c1b)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
Or if supported by the container library, pass -D__SANITIZER_DISABLE_CONTAINER_OVERFLOW__ to the compiler to disable instrumentation.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1405a55c8) in testing::internal::CmpHelperEQ>, class std::basic_string_view>>(char const *, char const *, class std::basic_string_view> const &, class std::basic_string_view> const &)
Shadow bytes around the buggy address:
0x1298940ced80: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
0x1298940cee00: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
0x1298940cee80: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
0x1298940cef00: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
0x1298940cef80: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 03
=>0x1298940cf000: fa fa 00 00 00 00[01]fc fa fa 00 00 00 00 00 03
0x1298940cf080: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
0x1298940cf100: fa fa fd fd fd fd fd fa fa fa fa fa fa fa fa fa
0x1298940cf180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x1298940cf200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x1298940cf280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb

NOTE: the stack trace above identifies the code that *accessed* the poisoned memory.
HINT: To identify the code that *poisoned* the memory, try the experimental setting ASAN_OPTIONS=poison_history_size=.
==6460==ABORTING
```

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the listed Visual Studio AddressSanitizer setup and the ctest filter for arrow-utility-test. Start by locating TestSecureString_AssertSecurelyCleared_Test and IsSecurelyCleared in the C++ test sources, then verify that the test completes without the reported container-overflow under AddressSanitizer.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.