TSAN check failed - sanitizer_deadlock_detector

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023-03-21T18:40:27.629Z] ThreadSanitizer: CHECK failed: sanitizer_deadlock_detector.h:67 "((n_all_locks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_contexts_)[0]))))" (0x40, 0x40) (tid=3912)
[2023-03-21T18:40:27.629Z]     #0 __tsan::CheckUnwind() /llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:386:25 (gtests_dbms+0x277fc4d)
[2023-03-21T18:40:27.629Z]     #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:86:5 (gtests_dbms+0x270b7d4)
[2023-03-21T18:40:27.629Z]     #2 __sanitizer::DeadlockDetectorTLS<__sanitizer::TwoLevelBitVector<1ul, __sanitizer::BasicBitVector<unsigned long> > >::addLock(unsigned long, unsigned long, unsigned int) /llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector.h:67:5 (gtests_dbms+0x26ff2f6)
[2023-03-21T18:40:27.629Z]     #3 onLockAfter /llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector.h:216:11 (gtests_dbms+0x26fe7ed)
[2023-03-21T18:40:27.629Z]     #4 __sanitizer::DD::MutexAfterLock(__sanitizer::DDCallback*, __sanitizer::DDMutex*, bool, bool) /llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector1.cpp:169:6 (gtests_dbms+0x26fe7ed)
[2023-03-21T18:40:27.629Z]     #5 __tsan::MutexPostLock(__tsan::ThreadState*, unsigned long, unsigned long, unsigned int, int) /llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cpp:197:14 (gtests_dbms+0x278ce80)
[2023-03-21T18:40:27.630Z]     #6 pthread_mutex_lock /llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:4255:5 (gtests_dbms+0x274170f)
[2023-03-21T18:40:27.630Z]     #7 std::__1::recursive_mutex::lock() <null> (libc++.so.1+0x67025)
[2023-03-21T18:40:27.630Z]     #8 std::__1::unique_lock<std::__1::recursive_mutex>::unique_lock(std::__1::recursive_mutex&) /usr/local/bin/../include/c++/v1/__mutex_base:118:61 (gtests_dbms+0x6ee33ff)
[2023-03-21T18:40:27.630Z]     #9 DB::DM::DeltaValueSpace::getLock() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.h:145:14 (gtests_dbms+0x6ee33ff)
[2023-03-21T18:40:27.630Z]     #10 DB::DM::Segment::getUpdateLock() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.h:520:63 (gtests_dbms+0x6ee33ff)
[2023-03-21T18:40:27.630Z]     #11 DB::DM::Segment::mustGetUpdateLock() const /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/Segment.h:524:25 (gtests_dbms+0x6ee33ff)
[2023-03-21T18:40:27.630Z]     #12 DB::DM::DeltaMergeStore::segmentMerge(DB::DM::DMContext&, std::__1::vector<std::__1::shared_ptr<DB::DM::Segment>, std::__1::allocator<std::__1::shared_ptr<DB::DM::Segment> > > const&, DB::DM::DeltaMergeStore::SegmentMergeReason) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp:307:37 (gtests_dbms+0xb1ea0db)
[2023-03-21T18:40:27.630Z]     #13 DB::DM::DeltaMergeStore::gcTrySegmentMerge(std::__1::shared_ptr<DB::DM::DMContext> const&, std::__1::shared_ptr<DB::DM::Segment> const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalBg.cpp:598:24 (gtests_dbms+0xb1d988d)
[2023-03-21T18:40:27.630Z]     #14 DB::DM::DeltaMergeStore::onSyncGc(long, DB::DM::GCOptions const&) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalBg.cpp:804:27 (gtests_dbms+0xb1dbcdf)
[2023-03-21T18:40:27.630Z]     #15 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:283:12 (gtests_dbms+0x6e422f2)
[2023-03-21T18:40:27.630Z]     #16 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb472705)
[2023-03-21T18:40:27.630Z]     #17 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb472705)
[2023-03-21T18:40:27.630Z]     #18 testing::Test::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xb44c2a5)
[2023-03-21T18:40:27.630Z]     #19 testing::TestInfo::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xb44dad4)
[2023-03-21T18:40:27.630Z]     #20 testing::TestCase::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xb44e79d)
[2023-03-21T18:40:27.630Z]     #21 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xb45e950)
[2023-03-21T18:40:27.630Z]     #22 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xb473ba5)
[2023-03-21T18:40:27.630Z]     #23 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xb473ba5)
[2023-03-21T18:40:27.630Z]     #24 testing::UnitTest::Run() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xb45dc38)
[2023-03-21T18:40:27.630Z]     #25 RUN_ALL_TESTS() /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x76ae7e7)
[2023-03-21T18:40:27.630Z]     #26 main /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:111:16 (gtests_dbms+0x76ae7e7)
[2023-03-21T18:40:27.630Z]     #27 __libc_start_main <null> (libc.so.6+0x22554)
[2023-03-21T18:40:27.630Z]     #28 _start <null> (gtests_dbms+0x26f9fe8)
[2023-03-21T18:40:27.630Z] 

4. What is your TiFlash version? (Required)

Start by reproducing StoreIngestTest_ConcurrentIngestAndWrite_Test in dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp under TSAN. Read sanitizer_deadlock_detector.h:67 and the stack frames in DeltaMergeStore_InternalSegment.cpp and DeltaMergeStore_InternalBg.cpp to trace the lock path. Done means the test no longer triggers the sanitizer_deadlock_detector check.