pingcap / pingcap/tiflash

ASAN/TSan failed compute test case

Open
#8,285 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/compute priority/P2 severity/minor type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Use 2023-10-31 latest version, error message:
[2023-10-31T18:37:38.841Z] WARNING: ThreadSanitizer: data race (pid=20500)
[2023-10-31T18:37:38.841Z] Read of size 8 at 0x7b6000030208 by main thread:
[2023-10-31T18:37:38.841Z] #0 memcpy /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x2ba6ce0)
[2023-10-31T18:37:38.841Z] #1 core::mem::maybe_uninit::MaybeUninit$LT$T$GT$::assume_init::h2f3b2e6915ec01bd /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/mem/maybe_uninit.rs:627:38 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #2 core::ptr::read::h5d271dd8cc4a87b2 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ptr/mod.rs:1140:9 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #3 alloc::vec::Vec$LT$T$C$A$GT$::remove::h77cb7968e7379f6d /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:1522:23 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #4 backtrace::symbolize::gimli::Cache::mapping_for_lib::h8e9c09979b3359f8 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:312:29 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #5 backtrace::symbolize::gimli::resolve::_$u7b$$u7b$closure$u7d$$u7d$::hc070d62f1c0573e8 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:354:24 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #6 backtrace::symbolize::gimli::Cache::with_global::h9110e7656f265813 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:9 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #7 backtrace::symbolize::gimli::resolve::h56abb7b895e03eed /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:346:5 (gtests_dbms+0xfe05c16)
[2023-10-31T18:37:38.841Z] #8 DB::tests::ComputeServerRunner_aggWithColumnPrune_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/tests/gtest_compute_server.cpp:564:1 (gtests_dbms+0x312650a)
[2023-10-31T18:37:38.841Z] #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc2bd085)
[2023-10-31T18:37:38.841Z] #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc2bd085)
[2023-10-31T18:37:38.841Z] #11 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc296c25)
[2023-10-31T18:37:38.841Z] #12 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc298454)
[2023-10-31T18:37:38.841Z] #13 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc29911d)
[2023-10-31T18:37:38.841Z] #14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc2a92d0)
[2023-10-31T18:37:38.841Z] #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc2be525)
[2023-10-31T18:37:38.841Z] #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::)(), char const) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc2be525)
[2023-10-31T18:37:38.841Z] #17 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc2a85b8)
[2023-10-31T18:37:38.841Z] #18 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x5f0b9ee)
[2023-10-31T18:37:38.841Z] #19 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x5f0b9ee)
[2023-10-31T18:37:38.841Z]
[2023-10-31T18:37:38.841Z] Previous write of size 8 at 0x7b6000030208 by thread T1456:
[2023-10-31T18:37:38.841Z] [failed to restore the stack]
[2023-10-31T18:37:38.841Z]
[2023-10-31T18:37:38.841Z] Location is heap block of size 1024 at 0x7b6000030000 allocated by thread T1450:
[2023-10-31T18:37:38.841Z] #0 malloc /llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:655:5 (gtests_dbms+0x2b9ba2d)
[2023-10-31T18:37:38.841Z] #1 alloc::alloc::alloc::h9f5d2080c32a5558 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:95:14 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #2 alloc::alloc::Global::alloc_impl::h2b2278f322e264e8 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:177:73 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #3 $LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$::allocate::h22ac9faea77a7755 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/alloc.rs:237:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #4 alloc::raw_vec::RawVec$LT$T$C$A$GT$::allocate_in::hd28f1e60fbe07f7c /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/raw_vec.rs:185:45 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #5 alloc::raw_vec::RawVec$LT$T$C$A$GT$::with_capacity_in::h73c2188b6b63f3ac /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/raw_vec.rs:131:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #6 alloc::vec::Vec$LT$T$C$A$GT$::with_capacity_in::hfdce392da93ae143 /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:673:20 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #7 alloc::vec::Vec$LT$T$GT$::with_capacity::hb875a34790c246fd /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/vec/mod.rs:483:9 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #8 backtrace::symbolize::gimli::Cache:🆕:h0b35aa2a5cc50baa /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:247:23 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #9 backtrace::symbolize::gimli::Cache::with_global::
$u7b$$u7b$closure$u7d$$u7d$::h917b603032c4e945 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:48 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #10 core::option::Option$LT$T$GT$::get_or_insert_with::h0869f117ab9c441a /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/option.rs:1546:49 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #11 backtrace::symbolize::gimli::Cache::with_global::h9110e7656f265813 /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:266:11 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #12 backtrace::symbolize::gimli::resolve::h56abb7b895e03eed /root/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.64/src/symbolize/gimli.rs:346:5 (gtests_dbms+0xfe05749)
[2023-10-31T18:37:38.841Z] #13 DB::getExceptionMessage(DB::Exception const&, bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/Exception.cpp:217:74 (gtests_dbms+0xf8c1f5a)
[2023-10-31T18:37:38.841Z] #14 DB::getCurrentExceptionMessage(bool, bool) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/Exception.cpp:106:23 (gtests_dbms+0xf8c0962)
[2023-10-31T18:37:38.841Z] #15 DB::MPPTask::runImpl() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:602:30 (gtests_dbms+0xf054ec0)
[2023-10-31T18:37:38.841Z] #16 DB::MPPTask::run()::$_19::operator()() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Flash/Mpp/MPPTask.cpp:203:97 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.841Z] #17 decltype(static_castDB::MPPTask::run()::$_19&(fp)()) std::__1::__invokeDB::MPPTask::run()::$_19&(DB::MPPTask::run()::$_19&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #18 void std::__1::__invoke_void_return_wrapper<void, true>::__callDB::MPPTask::run()::$_19&(DB::MPPTask::run()::$_19&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #19 std::__1::__function::__alloc_func<DB::MPPTask::run()::$_19, std::__1::allocatorDB::MPPTask::run()::$_19, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #20 std::__1::__function::__func<DB::MPPTask::run()::$_19, std::__1::allocatorDB::MPPTask::run()::$_19, void ()>::operator()() /usr/local/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xf056f63)
[2023-10-31T18:37:38.842Z] #21 std::__1::__function::__value_func<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #22 std::__1::function<void ()>::operator()() const /usr/local/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #23 decltype(static_cast<std::__1::function<void ()> const&>(fp)()) std::__1::__invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #24 std::__1::invoke_result<std::__1::function<void ()> const&>::type std::__1::invoke<std::__1::function<void ()> const&>(std::__1::function<void ()> const&) /usr/local/bin/../include/c++/v1/__functional/invoke.h:93:12 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #25 auto std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)::operator()<>(auto&&...) const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Common/ThreadFactory.h:48:26 (gtests_dbms+0xf91ec5d)
[2023-10-31T18:37:38.842Z] #26 decltype(static_cast<std::__1::function<void ()>>(fp)()) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::function<void ()>&&) /usr/local/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xf91e8e7)
[2023-10-31T18:37:38.842Z] #27 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>(std::__1::tuple<std::__1::function<void ()>, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)>&, std::__1::__tuple_indices<>) /usr/local/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xf91e8e7)
[2023-10-31T18:37:38.842Z] #28 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<std::__1::function<void ()> >(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, std::__1::function<void ()>&&)::'lambda'(auto&&...)> >(void*) /usr/local/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xf91e8e7)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failing DB::tests::ComputeServerRunner_aggWithColumnPrune_Test in dbms/src/Flash/tests/gtest_compute_server.cpp and inspect the reported paths in dbms/src/Common/Exception.cpp and dbms/src/Flash/Mpp/MPPTask.cpp. Reproduce the test under ThreadSanitizer or AddressSanitizer, trace the race, and confirm the compute test completes without sanitizer errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.