pingcap / pingcap/tiflash

tiflash got crash in Debug compalition mode when async_grpc_server is disable

Open
#9,725 2 comments 0 reactions 1 assignee View on GitHub

@guo-shaoge is already working on this.

Since Dec 12, 2024.

component/compute impact/crash may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/moderate type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)
  1. use tiup to start a cluster with one or more tiflash
  2. build tiflash in Debug mode
  3. change config to disable profiles.default.enable_async_server
  4. run a simple query
2. What did you expect to see? (Required)

runs ok

3. What did you see instead (Required)

tiflash crash

[2024/12/12 15:17:46.794 +08:00] [ERROR] [BaseDaemon.cpp:559] ["
  0x5b631eb8201a\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+213696538]
                \tlibs/libdaemon/src/BaseDaemon.cpp:210
  0x7921814951d0\t<unknown symbol> [libc.so.6+250320]
  0x7921814ee3f4\t<unknown symbol> [libc.so.6+615412]
  0x792181495120\tgsignal [libc.so.6+250144]
  0x79218147c4c3\tabort [libc.so.6+148675]
  0x79218147c3df\t<unknown symbol> [libc.so.6+148447]
  0x79218148d177\t__assert_fail [libc.so.6+217463]
  0x5b6320e29e42\tDB::FlashService::EstablishMPPConnection(grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) [tiflash+250035778]
                \tdbms/src/Flash/FlashService.cpp:637
  0x5b632349e730\ttikvpb::Tikv::Service::Service()::$_53::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+290359088]
                \tbuild/clang17_debug/contrib/kvproto/cpp/kvproto/tikvpb.grpc.pb.cc:2237
  0x5b632349e6dc\tdecltype(std::declval<tikvpb::Tikv::Service::Service()::$_53&>()(std::declval<tikvpb::Tikv::Service*>(), std::declval<grpc::ServerContext*>(), std::declval<mpp::EstablishMPPConnectionRequest const*>(), std::declval<grpc::ServerWriter<mpp::MPPDataPacket>*>())) std::__1::__invoke[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290359004]
                \t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:340
  0x5b632349e660\tgrpc::Status std::__1::__invoke_void_return_wrapper<grpc::Status, false>::__call[abi:ue170006]<tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*>(tikvpb::Tikv::Service::Service()::$_53&, tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358880]
                \t/usr/local/bin/../include/c++/v1/__type_traits/invoke.h:407
  0x5b632349e610\tstd::__1::__function::__alloc_func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290358800]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:192
  0x5b632349da24\tstd::__1::__function::__func<tikvpb::Tikv::Service::Service()::$_53, std::__1::allocator<tikvpb::Tikv::Service::Service()::$_53>, grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) [tiflash+290355748]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:363
  0x5b6323541ce5\tstd::__1::__function::__value_func<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()[abi:ue170006](tikvpb::Tikv::Service*&&, grpc::ServerContext*&&, mpp::EstablishMPPConnectionRequest const*&&, grpc::ServerWriter<mpp::MPPDataPacket>*&&) const [tiflash+291028197]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:517
  0x5b6323541c70\tstd::__1::function<grpc::Status (tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*)>::operator()(tikvpb::Tikv::Service*, grpc::ServerContext*, mpp::EstablishMPPConnectionRequest const*, grpc::ServerWriter<mpp::MPPDataPacket>*) const [tiflash+291028080]
                \t/usr/local/bin/../include/c++/v1/__functional/function.h:1168
  0x5b6323541c22\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()::operator()() const [tiflash+291028002]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:207
  0x5b63235402d4\tgrpc::Status grpc::internal::CatchingFunctionHandler<grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()>(grpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&)::'lambda'()&&) [tiflash+291021524]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:44
  0x5b632353fe35\tgrpc::internal::ServerStreamingHandler<tikvpb::Tikv::Service, mpp::EstablishMPPConnectionRequest, mpp::MPPDataPacket>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) [tiflash+291020341]
                \tcontrib/grpc/include/grpcpp/impl/codegen/method_handler.h:206
  0x5b632296cceb\tgrpc::Server::SyncRequest::ContinueRunAfterInterception() [tiflash+278621419]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:433
  0x5b632296c97f\tgrpc::Server::SyncRequest::Run(std::__1::shared_ptr<grpc::Server::GlobalCallbacks> const&, bool) [tiflash+278620543]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:421
  0x5b632296c4e3\tgrpc::Server::SyncRequestThreadManager::DoWork(void*, bool, bool) [tiflash+278619363]
                \tcontrib/grpc/src/cpp/server/server_cc.cc:791
  0x5b6322998be2\tgrpc::ThreadManager::MainWorkLoop() [tiflash+278801378]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:211
  0x5b632299890c\tgrpc::ThreadManager::WorkerThread::Run() [tiflash+278800652]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:44
  0x5b6322999469\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::operator()(void*) const [tiflash+278803561]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
  0x5b6322999449\tgrpc::ThreadManager::WorkerThread::WorkerThread(grpc::ThreadManager*)::$_0::__invoke(void*) [tiflash+278803529]
                \tcontrib/grpc/src/cpp/thread_manager/thread_manager.cc:36
  0x5b6323015e62\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const [tiflash+285605474]
                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:140
  0x5b6323015d49\tgrpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+285605193]
                \tcontrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
  0x7921814ec39d\t<unknown symbol> [libc.so.6+607133]"] [source=BaseDaemon] [thread_id=139]
4. What is your TiFlash version? (Required)

e3e6780e6dadaa44a86f71cc594b05f3c6ccc6df

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.