baidu / baidu/sofa-pbrpc

dead lock when stop rpc server

Open
#217 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.1k
Forks
649
PR merge metrics
No merged PRs in 30d

Description

发现一个dead lock场景:

stack 1:
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f845ed69dbd in _GI__pthread_mutex_lock (mutex=0x3df3c50) at ../nptl/pthread_mutex_lock.c:80
#2 0x00000000017d18a0 in sofa::pbrpc::RpcServerImpl::RestartListen() ()
#3 0x00000000017d275e in sofa::pbrpc::RpcServerImpl::TimerMaintain(boost::posix_time::ptime const&) ()
#4 0x00000000017baf3d in sofa::pbrpc::TimerWorker::on_timeout(boost::system::error_code const&) ()
#5 0x00000000017c0ca6 in void boost::asio::detail::strand_service::dispatch, boost::_bi::list2 >, boost::arg<1> > >, boost::system::error_code> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder1, boost::_bi::list2 >, boost::arg<1> > >, boost::system::error_code>&) ()
#6 0x00000000017c0f33 in void boost::asio::detail::wrapped_handler, boost::_bi::list2 >, boost::arg<1> > >, boost::asio::detail::is_continuation_if_running>::operator()(boost::system::error_code const&) ()
#7 0x00000000017c118e in boost::asio::detail::completion_handler, boost::_bi::list2 >, boost::arg<1> > >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, boost::_bi::bind_t, boost::_bi::list2 >, boost::arg<1> > > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#8 0x00000000017c1484 in void boost::asio::detail::strand_service::dispatch, boost::_bi::list2 >, boost::arg<1> > >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, boost::_bi::bind_t, boost::_bi::list2 >, boost::arg<1> > > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler, boost::_bi::list2 >, boost::arg<1> > >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code>, boost::_bi::bind_t, boost::_bi::list2 >, boost::arg<1> > > >&) ()
#9 0x00000000017c1805 in boost::asio::detail::wait_handler, boost::_bi::list2 >, boost::arg<1> > >, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#10 0x00000000017b1d71 in sofa::pbrpc::ThreadGroupImpl::thread_run(void*) ()
#11 0x00007f845ed676ba in start_thread (arg=0x7f84458c3700) at pthread_create.c:333
#12 0x00007f845e1fc3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

stack 2:
#0 0x00007f845ed6898d in pthread_join (threadid=140206079227648, thread_return=0x0) at pthread_join.c:90
#1 0x00000000017afc1b in sofa::pbrpc::ThreadGroupImpl::stop() ()
#2 0x00000000017d169e in sofa::pbrpc::RpcServerImpl::Stop() ()

RpcServerImpl::Stop逻辑:
加_start_stop_lock锁,调用 _maintain_thread_group->stop();
_maintain_thread_group->stop()需要join thread
RestartListen()正在进行,且wati在_start_stop_lock锁。
造成dead lock。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing RpcServerImpl::Stop(), ThreadGroupImpl::stop(), RestartListen(), and TimerMaintain(), focusing on _start_stop_lock and the maintain thread's join. Reproduce the shutdown path described by the two stacks and verify that stopping the RPC server cannot deadlock while timer maintenance is restarting the listener.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.