GRPC core when running tiflash random failpoint tests
Open
Nobody has claimed this yet.
affects-8.1
affects-8.5
component/compute
may-affects-5.4
may-affects-6.1
may-affects-6.5
may-affects-7.1
may-affects-7.5
severity/major
type/bug
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Run tiflash random failpoints test, enable "random_join_build_failpoint", tifalsh crashed with little possiblity:
0xaa1df4d faultSignalHandler(int, siginfo_t*, void*) [tiflash+178380621]
libs/libdaemon/src/BaseDaemon.cpp:221
0x7f1cadebf630 <unknown symbol> [libpthread.so.0+63024]
0x7f1cad902387 gsignal [libc.so.6+222087]
0x7f1cad903a78 __GI_abort [libc.so.6+227960]
0x7f1cad8fb1a6 __assert_fail_base [libc.so.6+192934]
0x7f1cad8fb252 __assert_fail [libc.so.6+193106]
0xc4509a1 grpc_core::RefCount::RefNonZero(grpc_core::DebugLocation const&, char const*) [tiflash+205851041]
contrib/grpc/src/core/lib/gprpp/ref_counted.h:137
0xc4508a5 grpc_stream_ref(grpc_stream_refcount*, char const*) [tiflash+205850789]
contrib/grpc/src/core/lib/transport/transport.h:88
0xc62c401 grpc_chttp2_stream_ref(grpc_chttp2_stream*, char const*) [tiflash+207799297]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:606
0xc62df71 grpc_chttp2_mark_stream_writable(grpc_chttp2_transport*, grpc_chttp2_stream*) [tiflash+207806321]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:928
0xc63210f read_action_locked(void*, grpc_error*) [tiflash+207823119]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2548
0xc49c693 grpc_combiner_continue_exec_ctx() [tiflash+206161555]
contrib/grpc/src/core/lib/iomgr/combiner.cc:236
0xc49fad2 grpc_core::ExecCtx::Flush() [tiflash+206174930]
contrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
0xc4a0778 grpc_core::Executor::RunClosures(char const*, grpc_closure_list) [tiflash+206178168]
contrib/grpc/src/core/lib/iomgr/executor.cc:130
0xc4a0a96 grpc_core::Executor::ThreadMain(void*) [tiflash+206178966]
contrib/grpc/src/core/lib/iomgr/executor.cc:248
0xc7c9906 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const [tiflash+209492230]
contrib/grpc/src/core/lib/gprpp/thd_posix.cc:140
0xc7c9815 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+209491989]
contrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
0x7f1cadeb7ea5 start_thread [libpthread.so.0+32421]"] [thread_id=1037]
another core stack:
[2022/08/31 14:58:20.695 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
0xaa891ddfaultSignalHandler(int, siginfo_t*, void*) [tiflash+178819549]
libs/libdaemon/src/BaseDaemon.cpp:221
0x7fd07bf9a630<unknown symbol> [libpthread.so.0+63024]
0x7fd07b9dd387gsignal [libc.so.6+222087]
0x7fd07b9dea78__GI_abort [libc.so.6+227960]
0x7fd07b9d61a6__assert_fail_base [libc.so.6+192934]
0x7fd07b9d6252__assert_fail [libc.so.6+193106]
0xc4b92e1grpc_core::RefCount::RefNonZero(grpc_core::DebugLocation const&, char const*) [tiflash+206279393]
contrib/grpc/src/core/lib/gprpp/ref_counted.h:137
0xc4b91e5grpc_stream_ref(grpc_stream_refcount*, char const*) [tiflash+206279141]
contrib/grpc/src/core/lib/transport/transport.h:88
0xc694d41grpc_chttp2_stream_ref(grpc_chttp2_stream*, char const*) [tiflash+208227649]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:606
0xc6968b1grpc_chttp2_mark_stream_writable(grpc_chttp2_transport*, grpc_chttp2_stream*) [tiflash+208234673]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:928
0xc69aa4fread_action_locked(void*, grpc_error*) [tiflash+208251471]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2548
0xc504fd3grpc_combiner_continue_exec_ctx() [tiflash+206589907]
contrib/grpc/src/core/lib/iomgr/combiner.cc:236
0xc508412grpc_core::ExecCtx::Flush() [tiflash+206603282]
contrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
0xc50ca9cpollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206621340]
contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
0xc50b393pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206615443]
contrib/grpc/src/core/lib/iomgr/ev_posix.cc:324
0xc517b8egrpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206666638]
contrib/grpc/src/core/lib/iomgr/pollset.cc:48
0xc538ffbcq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+206802939]
contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
0xc53854agrpc_completion_queue_next [tiflash+206800202]
contrib/grpc/src/core/lib/surface/completion_queue.cc:1114
0xc4783acgrpc_impl::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) [tiflash+206013356]
contrib/grpc/src/cpp/common/completion_queue_cc.cc:48
0x4737035grpc_impl::CompletionQueue::Next(void**, bool*) [tiflash+74674229]
contrib/grpc/include/grpcpp/impl/codegen/completion_queue_impl.h:176
0x47367e8DB::(anonymous namespace)::handleRpcs(grpc_impl::ServerCompletionQueue*, std::__1::shared_ptr<DB::Logger> const&) [tiflash+74672104]
dbms/src/Server/FlashGrpcServerHolder.cpp:42
0x473917fDB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashSecurityConfig&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_6::operator()() const [tiflash+74682751]
dbms/src/Server/FlashGrpcServerHolder.cpp:156```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the TiFlash random failpoint test with "random_join_build_failpoint" enabled and compare the two reported gRPC core stacks. Read contrib/grpc/src/core/lib/gprpp/ref_counted.h, the referenced chttp2 transport files, and dbms/src/Server/FlashGrpcServerHolder.cpp around lines 42 and 156. Done means the intermittent crash is explained and the relevant test no longer fails.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100