pingcap / pingcap/tiflash

UT: DeleteTiming/PageStorageFullGCConcurrentTest.DeletePage/BeforeFullGCPrepare deadlock

Open
#6,410 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/storage severity/minor type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
ninja gtests_dbms
./dbms/gtests_dbms
2. What did you expect to see? (Required)

All ut pass.

3. What did you see instead (Required)

hang on

[ RUN      ] DeleteTiming/PageStorageFullGCConcurrentTest.DeletePage/BeforeFullGCPrepare
[2022/12/02 18:50:34.527 +08:00] [WARN] [Context.cpp:1571] ["GlobalStoragePool has already been initialized."] [source=Context] [thread_id=1]
[2022/12/02 18:50:34.527 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig V3 {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_cached_fd_size: 100, blob_heavy_gc_valid_rate: 0.500, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.log] [thread_id=1]
[2022/12/02 18:50:34.527 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig V3 {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_cached_fd_size: 100, blob_heavy_gc_valid_rate: 0.500, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.data] [thread_id=1]
[2022/12/02 18:50:34.527 +08:00] [INFO] [PageStorageImpl.cpp:54] ["PageStorageImpl start. Config{ PageStorageConfig V3 {blob_file_limit_size: 268435456, blob_spacemap_type: 2, blob_cached_fd_size: 100, blob_heavy_gc_valid_rate: 0.500, blob_block_alignment_bytes: 0, wal_roll_size: 2097152, wal_max_persisted_log_files: 4} }"] [source=__global__.meta] [thread_id=1]
......
[2022/12/02 18:50:57.452 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=18002]
[2022/12/02 18:50:59.010 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=17982]
[2022/12/02 18:50:59.463 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=21673]
[2022/12/02 18:51:05.695 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=18624]
[2022/12/02 18:51:07.373 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=18617]
[2022/12/02 18:51:08.954 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=24578]
[2022/12/02 18:51:11.232 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=18616]
[2022/12/02 18:51:36.739 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=19970]
[2022/12/02 18:51:37.440 +08:00] [INFO] [PageDirectory.cpp:1620] ["After MVCC gc in memory [lowest_seq=27152] clean [invalid_snapshot_nums=0] [invalid_page_nums=0] [total_deref_counter=0] [all_del_entries=0]. Still exist [snapshot_nums=0], [page_nums=6]. Longest alive snapshot: [longest_alive_snapshot_time=0] [longest_alive_snapshot_seq=0] [stale_snapshot_nums=0]"] [source=__global__.meta] [thread_id=24591]
[2022/12/02 18:51:37.441 +08:00] [DEBUG] [Ctl.cpp:89] ["SYNC_FOR(before_PageStorageImpl::doGC_fullGC_prepare) trying... \n\n# Current Stack: \n       0xe6bf29e\tStackTrace::StackTrace() [gtests_dbms+241955486]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0xe712356\tDB::SyncPointCtl::sync(char const*) [gtests_dbms+242295638]\n                \tdbms/src/Common/SyncPoint/Ctl.cpp:89\n       0xd5b9229\tDB::PS::V3::PageStorageImpl::doGC(std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224105001]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:420\n       0xd5b8d54\tDB::PS::V3::PageStorageImpl::gcImpl(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+224103764]\n                \tdbms/src/Storages/Page/V3/PageStorageImpl.cpp:325\n       0x5f6bc0d\tDB::PageStorage::gc(bool, std::__1::shared_ptr<DB::WriteLimiter> const&, std::__1::shared_ptr<DB::ReadLimiter> const&) [gtests_dbms+100056077]\n                \tdbms/src/Storages/Page/PageStorage.h:198\n       0xd0e99e2\tDB::DM::GlobalStoragePool::gc(DB::Settings const&, bool, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [gtests_dbms+219060706]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:158\n       0xd0f3475\tDB::DM::GlobalStoragePool::restore()::$_9::operator()() const [gtests_dbms+219100277]\n                \tdbms/src/Storages/DeltaMerge/StoragePool.cpp:126\n       0xd0f3425\tdecltype(static_cast<DB::DM::GlobalStoragePool::restore()::$_9&>(fp)()) std::__1::__invoke<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100197]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xd0f33dd\tbool std::__1::__invoke_void_return_wrapper<bool, false>::__call<DB::DM::GlobalStoragePool::restore()::$_9&>(DB::DM::GlobalStoragePool::restore()::$_9&) [gtests_dbms+219100125]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/invoke.h:30\n       0xd0f33ad\tstd::__1::__function::__alloc_func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219100077]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:171\n       0xd0f24e9\tstd::__1::__function::__func<DB::DM::GlobalStoragePool::restore()::$_9, std::__1::allocator<DB::DM::GlobalStoragePool::restore()::$_9>, bool ()>::operator()() [gtests_dbms+219096297]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0xc6b64b2\tstd::__1::__function::__value_func<bool ()>::operator()() const [gtests_dbms+208364722]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:498\n       0xc6b6475\tstd::__1::function<bool ()>::operator()() const [gtests_dbms+208364661]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/__functional/function.h:1175\n       0xce9e8d5\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [gtests_dbms+216656085]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:226\n       0xce9f8cc\tDB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1::operator()() const [gtests_dbms+216660172]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:95\n       0xce9f875\tdecltype(static_cast<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(fp)()) std::__1::__invoke<DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1&&) [gtests_dbms+216660085]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/type_traits:3918\n       0xce9f855\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1>&, std::__1::__tuple_indices<>) [gtests_dbms+216660053]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:280\n       0xce9f5f2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [gtests_dbms+216659442]\n                \t/data1/qiuyang/projects/tiflash-env/sysroot/bin/../include/c++/v1/thread:291\n  0x7fb9e79d6ea5\tstart_thread [libpthread.so.0+32421]\n  0x7fb9e74e8b0d\tclone [libc.so.6+1043213]"] [source=SyncPointCtl] [thread_id=24591]
4. What is your TiFlash version? (Required)

master

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Build gtests_dbms with ninja, then run ./dbms/gtests_dbms and reproduce DeleteTiming/PageStorageFullGCConcurrentTest.DeletePage/BeforeFullGCPrepare. Start at PageStorageImpl.cpp:420 and the SyncPointCtl call in Ctl.cpp:89, then trace the GC path through StoragePool.cpp:158. Done means the test no longer hangs and all unit tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.