apache / apache/arrow

[CI][C++] arrow-dataset-dataset-writer-test in the ASAN UBSAN job failed

Open
#38,817 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Continuous Integration Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Since https://github.com/apache/arrow/commit/7df1cdd0fe364f86eca15bfc482210994ba8e2f0 but the change isn't related to this failure because it doesn't touch the dataset module.

https://github.com/apache/arrow/actions/runs/6919998294/job/18823986112#step:6:3047

```text
53/92 Test #56: arrow-dataset-dataset-writer-test ............***Failed 1.88 sec
Running arrow-dataset-dataset-writer-test, redirecting output into /build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.txt (attempt 1/1)
Running main() from ./googletest/src/gtest_main.cc
[==========] Running 20 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 20 tests from DatasetWriterTestFixture
[ RUN ] DatasetWriterTestFixture.Basic
[ OK ] DatasetWriterTestFixture.Basic (13 ms)
[ RUN ] DatasetWriterTestFixture.BasicFilePrefix
[ OK ] DatasetWriterTestFixture.BasicFilePrefix (3 ms)
[ RUN ] DatasetWriterTestFixture.BasicFileDirectoryPrefix
[ OK ] DatasetWriterTestFixture.BasicFileDirectoryPrefix (3 ms)
[ RUN ] DatasetWriterTestFixture.DirectoryCreateFails
[ OK ] DatasetWriterTestFixture.DirectoryCreateFails (2 ms)
[ RUN ] DatasetWriterTestFixture.MaxRowsOneWrite
[ OK ] DatasetWriterTestFixture.MaxRowsOneWrite (12 ms)
[ RUN ] DatasetWriterTestFixture.MaxRowsOneWriteWithFunctor
[ OK ] DatasetWriterTestFixture.MaxRowsOneWriteWithFunctor (8 ms)
[ RUN ] DatasetWriterTestFixture.MaxRowsOneWriteWithBrokenFunctor
[ OK ] DatasetWriterTestFixture.MaxRowsOneWriteWithBrokenFunctor (2 ms)
[ RUN ] DatasetWriterTestFixture.MaxRowsManyWrites
[ OK ] DatasetWriterTestFixture.MaxRowsManyWrites (11 ms)
[ RUN ] DatasetWriterTestFixture.MinRowGroup
[ OK ] DatasetWriterTestFixture.MinRowGroup (9 ms)
[ RUN ] DatasetWriterTestFixture.MaxRowGroup
[ OK ] DatasetWriterTestFixture.MaxRowGroup (8 ms)
[ RUN ] DatasetWriterTestFixture.MinAndMaxRowGroup
[ OK ] DatasetWriterTestFixture.MinAndMaxRowGroup (8 ms)
[ RUN ] DatasetWriterTestFixture.MinRowGroupBackpressure
[ OK ] DatasetWriterTestFixture.MinRowGroupBackpressure (34 ms)
[ RUN ] DatasetWriterTestFixture.ConcurrentWritesSameFile
[ OK ] DatasetWriterTestFixture.ConcurrentWritesSameFile (13 ms)
[ RUN ] DatasetWriterTestFixture.ConcurrentWritesDifferentFiles
[ OK ] DatasetWriterTestFixture.ConcurrentWritesDifferentFiles (18 ms)
[ RUN ] DatasetWriterTestFixture.MaxOpenFiles
[ OK ] DatasetWriterTestFixture.MaxOpenFiles (12 ms)
[ RUN ] DatasetWriterTestFixture.NoExistingDirectory
[ OK ] DatasetWriterTestFixture.NoExistingDirectory (4 ms)
[ RUN ] DatasetWriterTestFixture.DeleteExistingData
[ OK ] DatasetWriterTestFixture.DeleteExistingData (4 ms)
[ RUN ] DatasetWriterTestFixture.PartitionedDeleteExistingData
[ OK ] DatasetWriterTestFixture.PartitionedDeleteExistingData (4 ms)
[ RUN ] DatasetWriterTestFixture.LeaveExistingData
[ OK ] DatasetWriterTestFixture.LeaveExistingData (4 ms)
[ RUN ] DatasetWriterTestFixture.ErrOnExistingData
[ OK ] DatasetWriterTestFixture.ErrOnExistingData (1 ms)
[----------] 20 tests from DatasetWriterTestFixture (181 ms total)

[----------] Global test environment tear-down
[==========] 20 tests from 1 test suite ran. (182 ms total)
[ PASSED ] 20 tests.

=================================================================
==9995==ERROR: LeakSanitizer: detected memory leaks
#15 0x7fa79e9a497f in arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()::operator()() const /arrow/cpp/src/arrow/dataset/dataset_writer.cc:149:13
#16 0x7fa79e9a459b in std::enable_if<((!(std::is_void::value)) && (!(is_future::value))) && ((!(arrow::Future::is_empty)) || (std::is_same::value)), void>::type arrow::detail::ContinueFuture::operator(), std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&, arrow::Status, arrow::Future >(arrow::Future, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) const /arrow/cpp/src/arrow/util/future.h:150:23
#17 0x7fa79e9a4271 in void std::__invoke_impl&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#18 0x7fa79e9a3d75 in std::__invoke_result&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>::type std::__invoke&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#19 0x7fa79e9a3b5f in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#20 0x7fa79e9a3738 in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#21 0x7fa79e9a3606 in arrow::internal::FnOnce::FnImpl, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())> >::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42
#22 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17
#23 0x7fa7843f14b7 in arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce, arrow::StopToken, arrow::internal::FnOnce&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9
#24 0x7fa7843f12b6 in arrow::internal::FnOnce::FnImpl, arrow::StopToken, arrow::internal::FnOnce&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42
#25 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17
#26 0x7fa7843ea9c4 in arrow::internal::WorkerLoop(std::shared_ptr, std::_List_iterator) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11
#27 0x7fa7843e946b in arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7
#28 0x7fa7843e9293 in void std::__invoke_impl(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#29 0x7fa7843e9083 in std::__invoke_result::type std::__invoke(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14

Indirect leak of 17 byte(s) in 1 object(s) allocated from:
#0 0x55e00f5a4f1d in operator new(unsigned long) (/build/cpp/debug/arrow-dataset-dataset-writer-test+0x398f1d) (BuildId: 176377c2697ecb6a7274ce334f7231a363068218)
#1 0x7fa7990820e2 in void std::__cxx11::basic_string, std::allocator >::_M_construct(char*, char*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14
#2 0x7fa78bd03cb4 in arrow::fs::internal::MockFileSystem::Impl::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&, bool, std::shared_ptr const&) /arrow/cpp/src/arrow/filesystem/mockfs.cc:398:44
#3 0x7fa78bd02c58 in arrow::fs::internal::MockFileSystem::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&, std::shared_ptr const&) /arrow/cpp/src/arrow/filesystem/mockfs.cc:735:17
#4 0x7fa78bb45b27 in arrow::fs::FileSystem::OpenOutputStream(std::__cxx11::basic_string, std::allocator > const&) /arrow/cpp/src/arrow/filesystem/filesystem.cc:250:10
#5 0x7fa79e9a51d6 in arrow::dataset::internal::(anonymous namespace)::OpenWriter(arrow::dataset::FileSystemDatasetWriteOptions const&, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&) /arrow/cpp/src/arrow/dataset/dataset_writer.cc:127:3
#6 0x7fa79e9a497f in arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()::operator()() const /arrow/cpp/src/arrow/dataset/dataset_writer.cc:149:13
#7 0x7fa79e9a459b in std::enable_if<((!(std::is_void::value)) && (!(is_future::value))) && ((!(arrow::Future::is_empty)) || (std::is_same::value)), void>::type arrow::detail::ContinueFuture::operator(), std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&, arrow::Status, arrow::Future >(arrow::Future, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) const /arrow/cpp/src/arrow/util/future.h:150:23
#8 0x7fa79e9a4271 in void std::__invoke_impl&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(std::__invoke_other, arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#9 0x7fa79e9a3d75 in std::__invoke_result&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>::type std::__invoke&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&>(arrow::detail::ContinueFuture&, arrow::Future&, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'()&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#10 0x7fa79e9a3b5f in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::__call(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:420:11
#11 0x7fa79e9a3738 in void std::_Bind, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:503:17
#12 0x7fa79e9a3606 in arrow::internal::FnOnce::FnImpl, arrow::dataset::internal::(anonymous namespace)::DatasetWriterFileQueue::Start(arrow::util::AsyncTaskScheduler*, std::__cxx11::basic_string, std::allocator > const&)::'lambda'()::operator()() const::'lambda'())> >::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42
#13 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17
#14 0x7fa7843f14b7 in arrow::internal::ThreadPool::SpawnReal(arrow::internal::TaskHints, arrow::internal::FnOnce, arrow::StopToken, arrow::internal::FnOnce&&)::$_0::operator()() /arrow/cpp/src/arrow/util/thread_pool.cc:634:9
#15 0x7fa7843f12b6 in arrow::internal::FnOnce::FnImpl, arrow::StopToken, arrow::internal::FnOnce&&)::$_0>::invoke() /arrow/cpp/src/arrow/util/functional.h:152:42
#16 0x7fa7843c42e6 in arrow::internal::FnOnce::operator()() && /arrow/cpp/src/arrow/util/functional.h:140:17
#17 0x7fa7843ea9c4 in arrow::internal::WorkerLoop(std::shared_ptr, std::_List_iterator) /arrow/cpp/src/arrow/util/thread_pool.cc:457:11
#18 0x7fa7843e946b in arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5::operator()() const /arrow/cpp/src/arrow/util/thread_pool.cc:618:7
#19 0x7fa7843e9293 in void std::__invoke_impl(std::__invoke_other, arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#20 0x7fa7843e9083 in std::__invoke_result::type std::__invoke(arrow::internal::ThreadPool::LaunchWorkersUnlocked(int)::$_5&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
#21 0x7fa7843e8fd1 in void std::thread::_Invoker >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13
#22 0x7fa7843e8ea0 in std::thread::_Invoker >::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11
#23 0x7fa7843e8373 in std::thread::_State_impl > >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13
#5 0x7fa77805e252 in

Indirect leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x55e00f5a4f1d in operator new(unsigned long) (/build/cpp/debug/arrow-dataset-dataset-writer-test+0x398f1d) (BuildId: 176377c2697ecb6a7274ce334f7231a363068218)
#1 0x55e00f6689a0 in __gnu_cxx::new_allocator >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0x55e00f66882c in std::allocator_traits > >::allocate(std::allocator >&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20
#3 0x55e00f6684c4 in std::_Vector_base, std::allocator > >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:346:20
#4 0x55e00f667a2e in void std::vector, std::allocator > >::_M_range_initialize const*>(std::shared_ptr const*, std::shared_ptr const*, std::forward_iterator_tag) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:1582:14
#5 0x55e00f5b9097 in std::vector, std::allocator > >::vector(std::initializer_list >, std::allocator > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:629:2
#6 0x55e00f7677ab in arrow::dataset::internal::DatasetWriterTestFixture::SetUp() /arrow/cpp/src/arrow/dataset/dataset_writer_test.cc:73:22
#7 0x7fa7994c98ee in void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) (/usr/local/lib/libarrow_testing.so.1500+0xd608ee) (BuildId: 4068fe9148ba0adf20466726aadbd4ac18ef4795)

SUMMARY: AddressSanitizer: 3603 byte(s) leaked in 43 allocation(s).
Test had memory leaks. Editing XML
sed: can't read /build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.xml: No such file or directory
mv: cannot stat '/build/cpp/build/test-logs/arrow-dataset-dataset-writer-test.xml.bak': No such file or directory
/build/cpp/src/arrow/dataset
```

### Component(s)

C++, Continuous Integration

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.