tiflash got crash in Debug compalition mode when async_grpc_server is disable
Open
@guo-shaoge is already working on this.
Since Dec 12, 2024.
component/compute
impact/crash
may-affects-5.4
may-affects-6.1
may-affects-6.5
may-affects-7.1
may-affects-7.5
may-affects-8.1
may-affects-8.5
severity/moderate
type/bug
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
- use tiup to start a cluster with one or more tiflash
- build tiflash in Debug mode
- change config to disable
profiles.default.enable_async_server - run a simple query
2. What did you expect to see? (Required)
runs ok
3. What did you see instead (Required)
tiflash crash
[2024/12/12 15:17:46.794 +08:00] [ERROR] [BaseDaemon.cpp:559] ["
0x5b631eb8201a\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+213696538]
\tlibs/libdaemon/src/BaseDaemon.cpp:210
0x7921814951d0\t<unknown symbol> [libc.so.6+250320]
0x7921814ee3f4\t<unknown symbol> [libc.so.6+615412]
0x792181495120\tgsignal [libc.so.6+250144]
0x79218147c4c3\tabort [libc.so.6+148675]
0x79218147c3df\t<unknown symbol> [libc.so.6+148447]
0x79218148d177\t__assert_fail [libc.so.6+217463]
0x5b6320e29e42\tDB::FlashService::EstablishMPPConnection(grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) [tiflash+250035778]
\tdbms/src/Flash/FlashService.cpp:637
0x5b632349e730\ttikvpb::Tikv::Service::Service()::$_53::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+290359088]
\tbuild/clang17_debug/contrib/kvproto/cpp/kvproto/tikvpb.grpc.pb.cc:2237
0x5b632349e6dc\tdecltype(std::declval<tikvpb::Tikv::Service::Service()::$_53&>()(std::declval<tikvpb::Tikv::Service*>(), std::declval<grpc::ServerContext*>(), std::declval<mpp::EstablishMPPConnectionRequest const*>(), std::declval<grpc::ServerWriter<mpp::MPPDataPacket>*>())) std::__1::__invoke[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290359004]
\t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:340
0x5b632349e660\tgrpc::Status std::__1::__invoke_void_return_wrapper<grpc::Status, false>::__call[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358880]
\t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:407
0x5b632349e610\tstd::__1::__function::__alloc_func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358800]
\t/usr/local/bin/../include/c++/v1/__functional/function.h:192
0x5b632349da24\tstd::__1::__function::__func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290355748]
\t/usr/local/bin/../include/c++/v1/__functional/function.h:363
0x5b6323541ce5\tstd::__1::__function::__value_func<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) const [tiflash+291028197]
\t/usr/local/bin/../include/c++/v1/__functional/function.h:517
0x5b6323541c70\tstd::__1::function<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+291028080]
\t/usr/local/bin/../include/c++/v1/__functional/function.h:1168
0x5b6323541c22\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()::operator()() const [tiflash+291028002]
\tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:207
0x5b63235402d4\tgrpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) [tiflash+291021524]
\tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:44
0x5b632353fe35\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+291020341]
\tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:206
0x5b632296cceb\tgrpc::Server::SyncRequest::ContinueRunAfterInterception() [tiflash+278621419]
\tcontrib/grpc/src/cpp/server/server_cc.cc:433
0x5b632296c97f\tgrpc::Server::SyncRequest::Run(std::__1::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [tiflash+278620543]
\tcontrib/grpc/src/cpp/server/server_cc.cc:421
0x5b632296c4e3\tgrpc::Server::SyncRequestThreadManager::DoWork(void*, bool, bool) [tiflash+278619363]
\tcontrib/grpc/src/cpp/server/server_cc.cc:791
0x5b6322998be2\tgrpc::ThreadManager::MainWorkLoop() [tiflash+278801378]
\tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:211
0x5b632299890c\tgrpc::ThreadManager::WorkerThread::Run() [tiflash+278800652]
\tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:44
0x5b6322999469\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::operator()(void*) const [tiflash+278803561]
\tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
0x5b6322999449\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+278803529]
\tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
0x5b6323015e62\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const [tiflash+285605474]
\tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:140
0x5b6323015d49\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+285605193]
\tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
0x7921814ec39d\t<unknown symbol> [libc.so.6+607133]"] [source=BaseDaemon] [thread_id=139]
4. What is your TiFlash version? (Required)
e3e6780e6dadaa44a86f71cc594b05f3c6ccc6df
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.