tiflash meets crash

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. tiflash meet crash when running nto and tpcc workload

2. What did you expect to see? (Required)

no error

3. What did you see instead (Required)

error before crash:

[2023/03/03 04:29:48.332 +08:00] [ERROR] [BaseDaemon.cpp:376] [########################################] [source=BaseDaemon] [thread_id=11737]
[2023/03/03 04:29:48.333 +08:00] [ERROR] [BaseDaemon.cpp:377] ["(from thread 582) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=11737]
[2023/03/03 04:29:48.333 +08:00] [ERROR] [BaseDaemon.cpp:405] ["Address: NULL pointer."] [source=BaseDaemon] [thread_id=11737]
[2023/03/03 04:29:48.333 +08:00] [ERROR] [BaseDaemon.cpp:413] ["Access: read."] [source=BaseDaemon] [thread_id=11737]
[2023/03/03 04:29:48.333 +08:00] [ERROR] [BaseDaemon.cpp:425] ["Unknown si_code."] [source=BaseDaemon] [thread_id=11737]
[2023/03/03 04:29:48.333 +08:00] [ERROR] [BaseDaemon.cpp:569] ["\n 0x7166471\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+118908017]\n \tlibs/libdaemon/src/BaseDaemon.cpp:220\n 0x7fb50e04cd90\t [libc.so.6+347536]\n 0x83a9146\tcq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+138056006]\n \tcontrib/grpc/src/core/lib/surface/completion_queue.cc:999\n 0x1bface9\tDB::(anonymous namespace)::handleRpcs(grpc::ServerCompletionQueue*, std::__1::shared_ptrDB::Logger const&) [tiflash+29338857]\n \tdbms/src/Server/FlashGrpcServerHolder.cpp:50\n 0x1bfa8cd\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, std::__1::thread DB::ThreadFactory::newThread<DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_5>(bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, DB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashRaftConfig const&, std::__1::shared_ptrDB::Logger const&)::$_5&&)::'lambda'(auto&&...)> >(void*) [tiflash+29337805]\n \t/usr/local/bin/../include/c++/v1/thread:291\n 0x7fb50e097802\tstart_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=11737]

4. What is your TiFlash version? (Required)

TiFlash
Release Version: v6.7.0-alpha
Edition: Community
Git Commit Hash: fbed3eb9b09691015490ce1fd08254c309d0a1f8
Git Branch: heads/refs/tags/v6.7.0-alpha
UTC Build Time: 2023-02-24 11:34:52
Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto
Profile: RELWITHDEBINFO

Raft Proxy
Git Commit Hash: 9f3377b1dd390e9db141594f94a15064b456b0d4
Git Commit Branch: HEAD
UTC Build Time: 2023-02-24 11:41:15
Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: Unknown (env var does not exist when building)

Start by reproducing the crash with the nto and tpcc workloads on TiFlash v6.7.0-alpha, then inspect dbms/src/Server/FlashGrpcServerHolder.cpp around handleRpcs and contrib/grpc/src/core/lib/surface/completion_queue.cc:999. Use the provided segmentation-fault stack trace to narrow the failing gRPC completion-queue path. Done means the workloads complete without the reported crash.