pingcap / pingcap/tiflash

Asan test failed

Open
#10,478 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

severity/minor type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
2. What did you expect to see? (Required)
3. What did you see instead (Required)
[2025-10-12T19:11:53.533Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:86] ["SYNC_FOR(in_FastAddPeerImplWrite::after_write_segments) trying... \n\n# Current Stack: \n  0x55844102d395\t___interceptor_backtrace [gtests_dbms+124142485]\n                \t/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4355\n  0x55846582cae3\tStackTrace::StackTrace() [gtests_dbms+736508643]\n                \tdbms/src/Common/StackTrace.cpp:23\n  0x5584659ce472\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+738219122]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:86\n  0x55845ad088e5\tDB::FastAddPeerImplWrite(DB::TMTContext&, DB::TiFlashRaftProxyHelper const*, unsigned long, unsigned long, std::__1::tuple<std::__1::shared_ptr<DB::CheckpointInfo>, std::__1::shared_ptr<DB::Region>, raft_serverpb::RaftApplyState, raft_serverpb::RegionLocalState>&&, unsigned long) [gtests_dbms+557054181]\n                \tdbms/src/Storages/KVStore/MultiRaft/Disagg/FastAddPeer.cpp:460\n  0x55845ad0ca1e\tDB::FastAddPeerImpl(std::__1::shared_ptr<DB::FastAddPeerContext>, DB::TMTContext&, DB::TiFlashRaftProxyHelper const*, unsigned long, unsigned long, unsigned long) [gtests_dbms+557070878]\n                \tdbms/src/Storages/KVStore/MultiRaft/Disagg/FastAddPeer.cpp:521\n  0x55845ad1a337\tstd::__1::__function::__func<FastAddPeer::$_5, std::__1::allocator<FastAddPeer::$_5>, DB::FastAddPeerRes ()>::operator()() [gtests_dbms+557126455]\n                \tdbms/src/Storages/KVStore/MultiRaft/Disagg/FastAddPeer.cpp:781\n  0x55844a0ca99e\tstd::__1::__packaged_task_func<std::__1::function<DB::FastAddPeerRes ()>, std::__1::allocator<std::__1::function<DB::FastAddPeerRes ()>>, DB::FastAddPeerRes ()>::operator()() [gtests_dbms+275782046]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517\n  0x55844a0cbbad\tstd::__1::packaged_task<DB::FastAddPeerRes ()>::operator()() [gtests_dbms+275786669]\n                \t/usr/local/bin/../include/c++/v1/future:1891\n  0x55844a0caede\tDB::AsyncTasks<unsigned long, std::__1::function<DB::FastAddPeerRes ()>, DB::FastAddPeerRes>::addTaskWithCancel(unsigned long, std::__1::function<DB::FastAddPeerRes ()>, std::__1::function<void ()>)::'lambda'()::operator()() const [gtests_dbms+275783390]\n                \tdbms/src/Storages/KVStore/Utils/AsyncTasks.h:300\n  0x558465946e95\tDB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) [gtests_dbms+737664661]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517\n  0x558465952e9c\tstd::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<bool DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false>>::scheduleImpl<bool>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(bool&&)::'lambda'()>, void ()>::operator()() [gtests_dbms+737713820]\n                \tdbms/src/Common/UniThreadPool.cpp:167\n  0x558465941ba2\tDB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) [gtests_dbms+737643426]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517\n  0x55846594af03\tvoid* std::__1::__thread_proxy[abi:ue170006]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) [gtests_dbms+737681155]\n                \tdbms/src/Common/UniThreadPool.cpp:167\n  0x55844108118b\tasan_thread_start(void*) [gtests_dbms+124486027]\n                \t/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:225\n  0x7fd74770aea5\tstart_thread [libpthread.so.0+32421]\n  0x7fd747003b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=9]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:92] ["SYNC_FOR(in_FastAddPeerImplWrite::after_write_segments) matched waitAndPause(), paused until calling next()..."] [source=SyncPointCtl] [thread_id=9]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:71] ["waitAndPause(in_FastAddPeerImplWrite::after_write_segments) finished"] [source=SyncPointCtl] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:77] ["next(in_FastAddPeerImplWrite::after_write_segments) trying..."] [source=SyncPointCtl] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:96] ["SYNC_FOR(in_FastAddPeerImplWrite::after_write_segments) done"] [source=SyncPointCtl] [thread_id=9]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:79] ["next(in_FastAddPeerImplWrite::after_write_segments) done"] [source=SyncPointCtl] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [FastAddPeer.cpp:480] ["Finish write FAP snapshot, log_count=0"] [source="FastAddPeer(region_id=1 keyspace=4294967295 table_id=31 new_peer_id=2333)"] [thread_id=9]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [DEBUG] [Ctl.cpp:50] ["Disabled: in_FastAddPeerImplWrite::after_write_segments"] [source=SyncPointCtl] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:41.763 +00:00] [INFO] [FastAddPeer.cpp:829] ["Fetch task, result=Ok new_peer_id=2333 region_id=1 elapsed=18"] [source=FastAddPeer] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.765 +00:00] [DEBUG] [FastAddPeerCache.h:76] ["ParsedCheckpointDataHolder destroyed path=/tiflash/tmp/55638/page/local_100"] [thread_id=1]
[2025-10-12T19:11:53.534Z] /home/jenkins/agent/workspace/tiflash-sanitizer-daily/tiflash/dbms/src/Storages/KVStore/tests/gtest_kvstore_fast_add_peer.cpp:952: Failure
[2025-10-12T19:11:53.534Z] Failed
[2025-10-12T19:11:53.534Z] Code: 0. DB::Exception: not meet
[2025-10-12T19:11:53.534Z] 
[2025-10-12T19:11:53.534Z] Stack trace:
[2025-10-12T19:11:53.534Z] 
[2025-10-12T19:11:53.534Z]   0x55844102d395	___interceptor_backtrace [gtests_dbms+124142485]
[2025-10-12T19:11:53.534Z]                 	/tiflash-release-8.3/release-centos7-llvm/dockerfiles/misc/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4355
[2025-10-12T19:11:53.534Z]   0x55846582cae3	StackTrace::StackTrace() [gtests_dbms+736508643]
[2025-10-12T19:11:53.534Z]                 	dbms/src/Common/StackTrace.cpp:23
[2025-10-12T19:11:53.534Z]   0x55844a09f88c	DB::tests::RegionKVStoreTestFAP_Cancel4_Test::TestBody() [gtests_dbms+275605644]
[2025-10-12T19:11:53.534Z]                 	dbms/src/Common/Exception.h:46
[2025-10-12T19:11:53.534Z]   0x55845af954f2	void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtests_dbms+559727858]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:2443
[2025-10-12T19:11:53.534Z]   0x55845af35b36	testing::Test::Run() [gtests_dbms+559336246]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:2517
[2025-10-12T19:11:53.534Z]   0x55845af3909b	testing::TestInfo::Run() [gtests_dbms+559349915]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:2693
[2025-10-12T19:11:53.534Z]   0x55845af3a750	testing::TestCase::Run() [gtests_dbms+559355728]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:2811
[2025-10-12T19:11:53.534Z]   0x55845af65c4b	testing::internal::UnitTestImpl::RunAllTests() [gtests_dbms+559533131]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:5177
[2025-10-12T19:11:53.534Z]   0x55845af97c32	bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtests_dbms+559737906]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:2443
[2025-10-12T19:11:53.534Z]   0x55845af64ee7	testing::UnitTest::Run() [gtests_dbms+559529703]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/src/gtest.cc:4786
[2025-10-12T19:11:53.534Z]   0x55844b1dd936	main [gtests_dbms+293685558]
[2025-10-12T19:11:53.534Z]                 	contrib/googletest/googletest/include/gtest/gtest.h:2341
[2025-10-12T19:11:53.534Z]   0x7fd746f27555	__libc_start_main [libc.so.6+140629]
[2025-10-12T19:11:53.534Z]   0x558440fe7629	<unknown symbol> [gtests_dbms+123856425]
[2025-10-12T19:11:53.534Z] 
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.766 +00:00] [INFO] [AsyncTasks.h:105] ["Pending 0 tasks when destructing"] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.766 +00:00] [INFO] [AsyncTasks.h:109] ["Finish finalize thread pool"] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.766 +00:00] [WARN] [Context.cpp:1799] ["Release GlobalUniversalPageStorage(WriteNode)."] [source=Context] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.767 +00:00] [DEBUG] [TiFlashTestEnv.cpp:288] ["Created bucket mockbucket"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=1]
[2025-10-12T19:11:53.534Z] [2025/10/12 19:11:51.767 +00:00] [WARN] [TiFlashTestEnv.cpp:336] ["DeleteBucket: :"] [source="bucket=mockbucket root=tiflash_ut/"] [thread_id=1]
[2025-10-12T19:11:53.534Z] [  FAILED  ] RegionKVStoreTestFAP.Cancel4 (11096 ms)
[2025-10-12T19:11:53.534Z] [----------] 1 test from RegionKVStoreTestFAP (11096 ms total)
[2025-10-12T19:11:53.534Z] 
[2025-10-12T19:11:53.534Z] [----------] Global test environment tear-down
[2025-10-12T19:11:53.534Z] [==========] 1 test from 1 test case ran. (11096 ms total)
[2025-10-12T19:11:53.534Z] [  PASSED  ] 0 tests.
[2025-10-12T19:11:53.534Z] [  FAILED  ] 1 test, listed below:
[2025-10-12T19:11:53.534Z] [  FAILED  ] RegionKVStoreTestFAP.Cancel4
[2025-10-12T19:11:53.534Z] 
[2025-10-12T19:11:53.534Z]  1 FAILED TEST
4. What is your TiFlash version? (Required)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dbms/src/Storages/KVStore/tests/gtest_kvstore_fast_add_peer.cpp at line 952 and the RegionKVStoreTestFAP.Cancel4 test. Review the FastAddPeer.cpp synchronization and cancellation paths referenced in the stack trace, then reproduce the ASAN failure. Done means identifying and fixing the cause of the DB::Exception: not meet failure and making Cancel4 pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.