pingcap / pingcap/tiflash

Received signal Aborted (6) because grpc assertion failed.

Open
#2,224 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

stack:

2879586 2021.06.21 03:45:06.442175 [ 2251376 ] <Error> grpc: /tmp/tzg/release-centos7/prepare-environments/grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc, line number : 290, log msg : assertion failed: subchannel_list() == p->subchannel_list_.get() || subchannel_list() == p->latest_pending_subchannel_list_.get()
2879630 2021.06.21 03:45:06.500475 [ 2251377 ] <Error> BaseDaemon: (from thread 2251376) Received signal Aborted (6).
2879736 2021.06.21 03:45:06.602214 [ 2251377 ] <Error> BaseDaemon: 0. /lib64/libc.so.6(gsignal+0x37) [0x7f689c5ce207]
2879737 2021.06.21 03:45:06.602295 [ 2251377 ] <Error> BaseDaemon: 1. /lib64/libc.so.6(abort+0x148) [0x7f689c5cf8f8]
2879768 2021.06.21 03:45:06.642060 [ 2251377 ] <Error> BaseDaemon: 2. bin/tiflash/tiflash() [0x831cb20]
2879802 2021.06.21 03:45:06.664835 [ 2251377 ] <Error> BaseDaemon: 3. bin/tiflash/tiflash() [0x826fc32]
2879803 2021.06.21 03:45:06.664889 [ 2251377 ] <Error> BaseDaemon: 4. bin/tiflash/tiflash(grpc_combiner_continue_exec_ctx()+0x86) [0x828f0b6]
2879804 2021.06.21 03:45:06.664908 [ 2251377 ] <Error> BaseDaemon: 5. bin/tiflash/tiflash(grpc_core::ExecCtx::Flush()+0x73) [0x822d0b3]
2879805 2021.06.21 03:45:06.664927 [ 2251377 ] <Error> BaseDaemon: 6. bin/tiflash/tiflash(grpc_core::Executor::RunClosures(char const*, grpc_closure_list)+0xa2) [0x822d362]
2879806 2021.06.21 03:45:06.664944 [ 2251377 ] <Error> BaseDaemon: 7. bin/tiflash/tiflash(grpc_core::Executor::ThreadMain(void*)+0x172) [0x822d542]
2879807 2021.06.21 03:45:06.664960 [ 2251377 ] <Error> BaseDaemon: 8. bin/tiflash/tiflash() [0x85a9043]
2879808 2021.06.21 03:45:06.664976 [ 2251377 ] <Error> BaseDaemon: 9. /lib64/libpthread.so.0(+0x7dd5) [0x7f689cc6edd5]

version:
5.0.1 + hotfix
branch: release5.0-hotfix-tiflash-patch3
commit: e9218d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the abort on release5.0-hotfix-tiflash-patch3 at commit e9218d and inspect the grpc pick_first.cc assertion shown in the stack. Trace the subchannel-list state through the grpc combiner execution path; done means identifying a reproducible cause and verifying a fix with a regression test or equivalent reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, grpc
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.