grpc internal abort in pressure test
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Steps that may reproduce the issue
- Prepare TPCH database, SF=10, tiflash replicas=2
- Don't analyze any tables before running queries.
- Parrellely run 20 TPCH queries, each one runs queries sequentially.
"grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc, line number: 687, log msg : server stream 1025 still included in list 1"
for (int i = 0; i < STREAM_LIST_COUNT; i++) {
if (GPR_UNLIKELY(included[i])) {
gpr_log(GPR_ERROR, "%s stream %d still included in list %d",
t->is_client ? "client" : "server", id, i);
abort();
}
}
stack
[2022/05/24 22:16:15.762 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
0x1e88ca1 faultSignalHandler(int, siginfo_t*, void*) [tiflash+32017569]
libs/libdaemon/src/BaseDaemon.cpp:221
0x7f7547b68630 <unknown symbol> [libpthread.so.0+63024]
0x7f75475ab387 raise [libc.so.6+222087]
0x7f75475aca78 abort [libc.so.6+227960]
0x7fa692d destroy_stream_locked(void*, grpc_error*) [tiflash+133851437]
contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:726
0x7e484fa grpc_combiner_continue_exec_ctx() [tiflash+132416762]
contrib/grpc/src/core/lib/iomgr/combiner.cc:236
0x7e50695 pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+132449941]
contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
0x7e83b4d cq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+132660045]
contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
0x1d26bc3 DB::handleRpcs(grpc_impl::ServerCompletionQueue*, Poco::Logger*) [tiflash+30567363]
dbms/src/Server/Server.cpp:528
0x1d74375 auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+30884725]
dbms/src/Common/wrapInvocable.h:36
0x1d74665 std::__1::packaged_task<void ()>::operator()() [tiflash+30885477]
/usr/local/bin/../include/c++/v1/future:2089
0x1d77d23 DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+30899491]
dbms/src/Common/DynamicThreadPool.cpp:101
0x1d77380 DB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+30897024]
dbms/src/Common/DynamicThreadPool.cpp:115
0x1d784eb void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+30901483]
/usr/local/bin/../include/c++/v1/thread:291
0x7f7547b60ea5 start_thread [libpthread.so.0+32421]"] [thread_id=755]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the TPCH SF=10 setup, two TiFlash replicas, no table analysis, and 20 parallel sequential-query clients. Start with contrib/grpc/src/core/ext/transport/chttp2/transport.cc around lines 687 and 726, then trace DB::handleRpcs in dbms/src/Server/Server.cpp. Done means the reported internal abort is explained and the same pressure test completes without it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, grpc
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100