GRpc No match found for server name cause TiFlash crash

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2023/12/13 03:09:58.571 +00:00] [ERROR] [MPPTask.cpp:469] ["task running meets error: Code: 49, e.displayText() = DB::Exception: Check tunnel != nullptr failed: tunnel4+10: Can't find task [446283641820020850,4] within 10s., e.what() = DB::Exception, Stack trace:\n\n\n       0x1c54d3e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+29707582]\n                \tdbms/src/Common/Exception.h:46\n       0x82636b6\tDB::GRPCReceiverContext::establishMPPConnectionLocalV2(DB::ExchangeRecvRequest const&, unsigned long, DB::LocalRequestHandler&, bool, bool) [tiflash+136722102]\n                \tdbms/src/Flash/Mpp/GRPCReceiverContext.cpp:296\n       0x1c14f84\tDB::ExchangeReceiverBase<DB::GRPCReceiverContext>::ExchangeReceiverBase(std::__1::shared_ptr<DB::GRPCReceiverContext>, unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, int, std::__1::vector<std::__1::tuple<std::__1::shared_ptr<mpp::DispatchTaskRequest>, std::__1::vector<pingcap::kv::RegionVerID, std::__1::allocator<pingcap::kv::RegionVerID> >, unsigned long>, std::__1::allocator<std::__1::tuple<std::__1::shared_ptr<mpp::DispatchTaskRequest>, std::__1::vector<pingcap::kv::RegionVerID, std::__1::allocator<pingcap::kv::RegionVerID> >, unsigned long> > > const&) [tiflash+29446020]\n                \tdbms/src/Flash/Mpp/ExchangeReceiver.cpp:338\n       0x827e991\tstd::__1::__function::__func<DB::MPPTask::initExchangeReceivers()::$_18, std::__1::allocator<DB::MPPTask::initExchangeReceivers()::$_18>, bool (tipb::Executor const&)>::operator()(tipb::Executor const&) [tiflash+136833425]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:345\n       0x8189682\tvoid DB::traverseExecutorTree<std::__1::function<bool (tipb::Executor const&)>&>(tipb::Executor const&, std::__1::function<bool (tipb::Executor const&)>&)::'lambda'(tipb::Executor const&)::operator()(tipb::Executor const&) const [tiflash+135829122]\n                \tdbms/src/Flash/Statistics/traverseExecutors.h:79\n       0x8189186\tvoid DB::traverseExecutors<std::__1::function<bool (tipb::Executor const&)> >(tipb::DAGRequest const*, std::__1::function<bool (tipb::Executor const&)>&&) [tiflash+135827846]\n                \tdbms/src/Flash/Statistics/traverseExecutors.h:94\n       0x8279898\tDB::MPPTask::runImpl() [tiflash+136812696]\n                \tdbms/src/Flash/Mpp/MPPTask.cpp:389\n       0x1d60168\tauto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+30802280]\n                \tdbms/src/Common/wrapInvocable.h:36\n       0x1c64c53\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+29772883]\n                \tdbms/src/Common/DynamicThreadPool.cpp:101\n       0x1c642b4\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+29770420]\n                \tdbms/src/Common/DynamicThreadPool.cpp:115\n       0x1c657c2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+29775810]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f1b09d5a802\tstart_thread [libc.so.6+653314]\n  0x7f1b09cfa450\t__GI___clone3 [libc.so.6+259152]"] [source="MPP<query:<query_ts:1702436988566420096, local_query_id:18097, server_id:1085732, start_ts:446283641820020850>,task_id:10>"] [thread_id=5707]
[2023/12/13 03:09:58.571 +00:00] [WARN] [MPPTaskManager.cpp:155] ["Begin to abort query: <query_ts:1702436988566420096, local_query_id:18097, server_id:1085732, start_ts:446283641820020850>, abort type: ONERROR, reason: From MPP<query:<query_ts:1702436988566420096, local_query_id:18097, server_id:1085732, start_ts:446283641820020850>,task_id:10>: Code: 49, e.displayText() = DB::Exception: Check tunnel != nullptr failed: tunnel4+10: Can't find task [446283641820020850,4] within 10s., e.what() = DB::Exception,"] [thread_id=5707]
[2023/12/13 03:09:58.571 +00:00] [WARN] [MPPTask.cpp:511] ["Begin abort task: MPP<query:<query_ts:1702436988566420096, local_query_id:18097, server_id:1085732, start_ts:446283641820020850>,task_id:10>, abort type: ONERROR"] [source="MPP<query:<query_ts:1702436988566420096, local_query_id:18097, server_id:1085732, start_ts:446283641820020850>,task_id:10>"] [thread_id=5707]
[2023/12/13 03:10:47.239 +00:00] [ERROR] [MPPTask.cpp:469] ["task running meets error: Code: 0, e.displayText() = DB::Exception: tunnel4+-1 is timeout, e.what() = DB::Exception, Stack trace:\n\n\n       0x1c54d3e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+29707582]\n                \tdbms/src/Common/Exception.h:46\n       0x829220e\tDB::MPPTunnel::waitUntilConnectedOrFinished(std::__1::unique_lock<std::__1::mutex>&) [tiflash+136913422]\n                \tdbms/src/Flash/Mpp/MPPTunnel.cpp:343\n       0x8291918\tDB::MPPTunnel::write(std::__1::shared_ptr<DB::TrackedMppDataPacket>&&) [tiflash+136911128]\n                \tdbms/src/Flash/Mpp/MPPTunnel.cpp:163\n       0x1c2406b\tDB::MPPTunnelSetBase<DB::MPPTunnel>::sendExecutionSummary(tipb::SelectResponse const&) [tiflash+29507691]\n                \tdbms/src/Flash/Mpp/MPPTunnelSet.cpp:39\n       0x827af34\tDB::MPPTask::runImpl() [tiflash+136818484]\n                \tdbms/src/Flash/Mpp/MPPTask.cpp:421\n       0x1d60168\tauto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+30802280]\n                \tdbms/src/Common/wrapInvocable.h:36\n       0x1c64c53\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+29772883]\n                \tdbms/src/Common/DynamicThreadPool.cpp:101\n       0x1c642b4\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+29770420]\n                \tdbms/src/Common/DynamicThreadPool.cpp:115\n       0x1c657c2\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+29775810]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f1b09d5a802\tstart_thread [libc.so.6+653314]\n  0x7f1b09cfa450\t__GI___clone3 [libc.so.6+259152]"] [source="MPP<query:<query_ts:1702436987230751199, local_query_id:17967, server_id:1085732, start_ts:446283641570984098>,task_id:4>"] [thread_id=6468]
[2023/12/13 03:10:47.239 +00:00] [WARN] [MPPTaskManager.cpp:155] ["Begin to abort query: <query_ts:1702436987230751199, local_query_id:17967, server_id:1085732, start_ts:446283641570984098>, abort type: ONERROR, reason: From MPP<query:<query_ts:1702436987230751199, local_query_id:17967, server_id:1085732, start_ts:446283641570984098>,task_id:4>: Code: 0, e.displayText() = DB::Exception: tunnel4+-1 is timeout, e.what() = DB::Exception,"] [thread_id=6468]
[2023/12/13 03:10:47.239 +00:00] [WARN] [MPPTask.cpp:511] ["Begin abort task: MPP<query:<query_ts:1702436987230751199, local_query_id:17967, server_id:1085732, start_ts:446283641570984098>,task_id:4>, abort type: ONERROR"] [source="MPP<query:<query_ts:1702436987230751199, local_query_id:17967, server_id:1085732, start_ts:446283641570984098>,task_id:4>"] [thread_id=6468]
[2023/12/13 04:11:31.597 +00:00] [ERROR] [BaseDaemon.cpp:376] [########################################] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.597 +00:00] [ERROR] [BaseDaemon.cpp:377] ["(from thread 5804) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.597 +00:00] [ERROR] [BaseDaemon.cpp:407] ["Address: 0x7f1970bfbb"] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.597 +00:00] [ERROR] [BaseDaemon.cpp:413] ["Access: read."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.597 +00:00] [ERROR] [BaseDaemon.cpp:422] ["Address not mapped to object."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.600 +00:00] [ERROR] [Server.cpp:379] ["/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/tsi/ssl_transport_security.cc, line number: 1874, log msg : No match found for server name: tc-tiflash-10.tc-tiflash-peer.comp-tiflash-tps-5131024-1-679.svc."] [source=grpc] [thread_id=5852]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:569] ["\n       0x7554c51\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+123030609]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:220\n  0x7f1b09d0fdb0\t<unknown symbol> [libc.so.6+347568]\n       0x8ccee29\tbio_ctrl [tiflash+147648041]\n                \tcontrib/boringssl/crypto/bio/pair.c:411\n       0x8ccd3f5\tBIO_pending [tiflash+147641333]\n                \tcontrib/boringssl/crypto/bio/bio.c:312\n       0x8c586ac\tssl_handshaker_next(tsi_handshaker*, unsigned char const*, unsigned long, unsigned char const**, unsigned long*, tsi_handshaker_result**, void (*)(tsi_result, void*, unsigned char const*, unsigned long, tsi_handshaker_result*), void*) [tiflash+147162796]\n                \tcontrib/grpc/src/core/tsi/ssl_transport_security.cc:1568\n       0x8b0568b\tgrpc_core::(anonymous namespace)::SecurityHandshaker::OnHandshakeDataReceivedFromPeerFn(void*, grpc_error*) [tiflash+145774219]\n                \tcontrib/grpc/src/core/lib/security/transport/security_handshaker.cc:463\n       0x8aa3141\tpollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+145371457]\n                \tcontrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1139\n       0x8b20fae\tcq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+145887150]\n                \tcontrib/grpc/src/core/lib/surface/completion_queue.cc:1047\n       0x1e8af99\tDB::(anonymous namespace)::handleRpcs(grpc::ServerCompletionQueue*, std::__1::shared_ptr<DB::Logger> const&) [tiflash+32026521]\n                \tdbms/src/Server/FlashGrpcServerHolder.cpp:52\n       0x1e8ab7d\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_5>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_5&&)::'lambda'(auto&&...)> >(void*) [tiflash+32025469]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f1b09d5a802\tstart_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:376] [########################################] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:377] ["(from thread 5852) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:407] ["Address: 0x4400"] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:413] ["Access: read."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.604 +00:00] [ERROR] [BaseDaemon.cpp:422] ["Address not mapped to object."] [source=BaseDaemon] [thread_id=6514]
[2023/12/13 04:11:31.609 +00:00] [ERROR] [BaseDaemon.cpp:569] ["\n       0x7554c51\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+123030609]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:220\n  0x7f1b09d0fdb0\t<unknown symbol> [libc.so.6+347568]\n       0x8c57503\tserver_handshaker_factory_alpn_callback(ssl_st*, unsigned char const**, unsigned char*, unsigned char const*, unsigned int, void*) [tiflash+147158275]\n                \tcontrib/grpc/src/core/tsi/ssl_transport_security.cc:1884\n       0x8c803ae\tbssl::ssl_negotiate_alpn(bssl::SSL_HANDSHAKE*, unsigned char*, ssl_early_callback_ctx const*) [tiflash+147325870]\n                \tcontrib/boringssl/ssl/extensions.cc:1570\n       0x8cbf426\tbssl::tls13_server_handshake(bssl::SSL_HANDSHAKE*) [tiflash+147584038]\n                \tcontrib/boringssl/ssl/tls13_server.cc:1242\n       0x8c9e8dc\tbssl::ssl_server_handshake(bssl::SSL_HANDSHAKE*) [tiflash+147450076]\n                \tcontrib/boringssl/ssl/handshake_server.cc:1835\n       0x8c9018a\tbssl::ssl_run_handshake(bssl::SSL_HANDSHAKE*, bool*) [tiflash+147390858]\n                \tcontrib/boringssl/ssl/handshake.cc:738\n       0x8caae2a\tSSL_do_handshake [tiflash+147500586]\n                \tcontrib/boringssl/ssl/ssl_lib.cc:841\n       0x8c58680\tssl_handshaker_next(tsi_handshaker*, unsigned char const*, unsigned long, unsigned char const**, unsigned long*, tsi_handshaker_result**, void (*)(tsi_result, void*, unsigned char const*, unsigned long, tsi_handshaker_result*), void*) [tiflash+147162752]\n                \tcontrib/grpc/src/core/tsi/ssl_transport_security.cc:1568\n       0x8b0568b\tgrpc_core::(anonymous namespace)::SecurityHandshaker::OnHandshakeDataReceivedFromPeerFn(void*, grpc_error*) [tiflash+145774219]\n                \tcontrib/grpc/src/core/lib/security/transport/security_handshaker.cc:463\n       0x8aa3141\tpollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+145371457]\n                \tcontrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1139\n       0x8b20fae\tcq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+145887150]\n                \tcontrib/grpc/src/core/lib/surface/completion_queue.cc:1047\n       0x1e8af99\tDB::(anonymous namespace)::handleRpcs(grpc::ServerCompletionQueue*, std::__1::shared_ptr<DB::Logger> const&) [tiflash+32026521]\n                \tdbms/src/Server/FlashGrpcServerHolder.cpp:52\n       0x1e8c05d\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_6>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_6&&)::'lambda'(auto&&...)> >(void*) [tiflash+32030813]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f1b09d5a802\tstart_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=6514]

similar case #7024 updated: not similar

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

v7.1.3

Start with the supplied logs and trace the failure through dbms/src/Flash/Mpp/GRPCReceiverContext.cpp, dbms/src/Flash/Mpp/MPPTunnel.cpp, and contrib/grpc/src/core/tsi/ssl_transport_security.cc. Then inspect dbms/src/Server/FlashGrpcServerHolder.cpp and the reported gRPC server-name mismatch; done means the matching failure no longer causes the TiFlash segmentation fault.