pingcap / pingcap/tiflash

GRPC core when running tiflash random failpoint tests

Open
#5,722 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-8.1 affects-8.5 component/compute may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 severity/major type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Run tiflash random failpoints test, enable "random_join_build_failpoint", tifalsh crashed with little possiblity:

       0xaa1df4d    faultSignalHandler(int, siginfo_t*, void*) [tiflash+178380621]
                    libs/libdaemon/src/BaseDaemon.cpp:221
  0x7f1cadebf630    <unknown symbol> [libpthread.so.0+63024]
  0x7f1cad902387    gsignal [libc.so.6+222087]
  0x7f1cad903a78    __GI_abort [libc.so.6+227960]
  0x7f1cad8fb1a6    __assert_fail_base [libc.so.6+192934]
  0x7f1cad8fb252    __assert_fail [libc.so.6+193106]
       0xc4509a1    grpc_core::RefCount::RefNonZero(grpc_core::DebugLocation const&, char const*) [tiflash+205851041]
                    contrib/grpc/src/core/lib/gprpp/ref_counted.h:137
       0xc4508a5    grpc_stream_ref(grpc_stream_refcount*, char const*) [tiflash+205850789]
                    contrib/grpc/src/core/lib/transport/transport.h:88
       0xc62c401    grpc_chttp2_stream_ref(grpc_chttp2_stream*, char const*) [tiflash+207799297]
                    contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:606
       0xc62df71    grpc_chttp2_mark_stream_writable(grpc_chttp2_transport*, grpc_chttp2_stream*) [tiflash+207806321]
                    contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:928
       0xc63210f    read_action_locked(void*, grpc_error*) [tiflash+207823119]
                    contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2548
       0xc49c693    grpc_combiner_continue_exec_ctx() [tiflash+206161555]
                    contrib/grpc/src/core/lib/iomgr/combiner.cc:236
       0xc49fad2    grpc_core::ExecCtx::Flush() [tiflash+206174930]
                    contrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
       0xc4a0778    grpc_core::Executor::RunClosures(char const*, grpc_closure_list) [tiflash+206178168]
                    contrib/grpc/src/core/lib/iomgr/executor.cc:130
       0xc4a0a96    grpc_core::Executor::ThreadMain(void*) [tiflash+206178966]
                    contrib/grpc/src/core/lib/iomgr/executor.cc:248
       0xc7c9906    grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const [tiflash+209492230]
                    contrib/grpc/src/core/lib/gprpp/thd_posix.cc:140
       0xc7c9815    grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) [tiflash+209491989]
                    contrib/grpc/src/core/lib/gprpp/thd_posix.cc:110
  0x7f1cadeb7ea5    start_thread [libpthread.so.0+32421]"] [thread_id=1037]


another core stack:
[2022/08/31 14:58:20.695 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
       0xaa891ddfaultSignalHandler(int, siginfo_t*, void*) [tiflash+178819549]
                libs/libdaemon/src/BaseDaemon.cpp:221
  0x7fd07bf9a630<unknown symbol> [libpthread.so.0+63024]
  0x7fd07b9dd387gsignal [libc.so.6+222087]
  0x7fd07b9dea78__GI_abort [libc.so.6+227960]
  0x7fd07b9d61a6__assert_fail_base [libc.so.6+192934]
  0x7fd07b9d6252__assert_fail [libc.so.6+193106]
       0xc4b92e1grpc_core::RefCount::RefNonZero(grpc_core::DebugLocation const&, char const*) [tiflash+206279393]
                contrib/grpc/src/core/lib/gprpp/ref_counted.h:137
       0xc4b91e5grpc_stream_ref(grpc_stream_refcount*, char const*) [tiflash+206279141]
                contrib/grpc/src/core/lib/transport/transport.h:88
       0xc694d41grpc_chttp2_stream_ref(grpc_chttp2_stream*, char const*) [tiflash+208227649]
                contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:606
       0xc6968b1grpc_chttp2_mark_stream_writable(grpc_chttp2_transport*, grpc_chttp2_stream*) [tiflash+208234673]
                contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:928
       0xc69aa4fread_action_locked(void*, grpc_error*) [tiflash+208251471]
                contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2548
       0xc504fd3grpc_combiner_continue_exec_ctx() [tiflash+206589907]
                contrib/grpc/src/core/lib/iomgr/combiner.cc:236
       0xc508412grpc_core::ExecCtx::Flush() [tiflash+206603282]
                contrib/grpc/src/core/lib/iomgr/exec_ctx.cc:156
       0xc50ca9cpollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206621340]
                contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
       0xc50b393pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206615443]
                contrib/grpc/src/core/lib/iomgr/ev_posix.cc:324
       0xc517b8egrpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+206666638]
                contrib/grpc/src/core/lib/iomgr/pollset.cc:48
       0xc538ffbcq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+206802939]
                contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
       0xc53854agrpc_completion_queue_next [tiflash+206800202]
                contrib/grpc/src/core/lib/surface/completion_queue.cc:1114
       0xc4783acgrpc_impl::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) [tiflash+206013356]
                contrib/grpc/src/cpp/common/completion_queue_cc.cc:48
       0x4737035grpc_impl::CompletionQueue::Next(void**, bool*) [tiflash+74674229]
                contrib/grpc/include/grpcpp/impl/codegen/completion_queue_impl.h:176
       0x47367e8DB::(anonymous namespace)::handleRpcs(grpc_impl::ServerCompletionQueue*, std::__1::shared_ptr<DB::Logger> const&) [tiflash+74672104]
                dbms/src/Server/FlashGrpcServerHolder.cpp:42
       0x473917fDB::FlashGrpcServerHolder::FlashGrpcServerHolder(DB::Context&, Poco::Util::LayeredConfiguration&, DB::TiFlashSecurityConfig&, DB::TiFlashRaftConfig const&, std::__1::shared_ptr<DB::Logger> const&)::$_6::operator()() const [tiflash+74682751]
                dbms/src/Server/FlashGrpcServerHolder.cpp:156```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the TiFlash random failpoint test with "random_join_build_failpoint" enabled and compare the two reported gRPC core stacks. Read contrib/grpc/src/core/lib/gprpp/ref_counted.h, the referenced chttp2 transport files, and dbms/src/Server/FlashGrpcServerHolder.cpp around lines 42 and 156. Done means the intermittent crash is explained and the relevant test no longer fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.