pingcap / pingcap/tiflash

grpc internal abort in pressure test

Open
#5,013 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 affects-7.1 affects-7.5 component/compute component/mpp may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 may-affects-5.4 may-affects-6.0 may-affects-6.1 severity/major type/bug
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Steps that may reproduce the issue

  1. Prepare TPCH database, SF=10, tiflash replicas=2
  2. Don't analyze any tables before running queries.
  3. Parrellely run 20 TPCH queries, each one runs queries sequentially.

"grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc, line number: 687, log msg : server stream 1025 still included in list 1"

for (int i = 0; i < STREAM_LIST_COUNT; i++) {
    if (GPR_UNLIKELY(included[i])) {
      gpr_log(GPR_ERROR, "%s stream %d still included in list %d",
              t->is_client ? "client" : "server", id, i);
      abort();
    }
  }

stack

[2022/05/24 22:16:15.762 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
       0x1e88ca1    faultSignalHandler(int, siginfo_t*, void*) [tiflash+32017569]
                    libs/libdaemon/src/BaseDaemon.cpp:221
  0x7f7547b68630    <unknown symbol> [libpthread.so.0+63024]
  0x7f75475ab387    raise [libc.so.6+222087]
  0x7f75475aca78    abort [libc.so.6+227960]
       0x7fa692d    destroy_stream_locked(void*, grpc_error*) [tiflash+133851437]
                    contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:726
       0x7e484fa    grpc_combiner_continue_exec_ctx() [tiflash+132416762]
                    contrib/grpc/src/core/lib/iomgr/combiner.cc:236
       0x7e50695    pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+132449941]
                    contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
       0x7e83b4d    cq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+132660045]
                    contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
       0x1d26bc3    DB::handleRpcs(grpc_impl::ServerCompletionQueue*, Poco::Logger*) [tiflash+30567363]
                    dbms/src/Server/Server.cpp:528
       0x1d74375    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+30884725]
                    dbms/src/Common/wrapInvocable.h:36
       0x1d74665    std::__1::packaged_task<void ()>::operator()() [tiflash+30885477]
                    /usr/local/bin/../include/c++/v1/future:2089
       0x1d77d23    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+30899491]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1d77380    DB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+30897024]
                    dbms/src/Common/DynamicThreadPool.cpp:115
       0x1d784eb    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+30901483]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f7547b60ea5    start_thread [libpthread.so.0+32421]"] [thread_id=755]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the TPCH SF=10 setup, two TiFlash replicas, no table analysis, and 20 parallel sequential-query clients. Start with contrib/grpc/src/core/ext/transport/chttp2/transport.cc around lines 687 and 726, then trace DB::handleRpcs in dbms/src/Server/Server.cpp. Done means the reported internal abort is explained and the same pressure test completes without it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.