apache / apache/incubator-pegasus

Pegasus Coredump at boost.asio epoll_reactor 1.11.5

Open
#387 5 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
If possible, provide a recipe for reproducing the error.

Normal bootstrap. c3srv-xiaoai, on replica-server (2019/08/29).

2. What did you expect to see?

No coredump.

3. What did you see instead?

Coredump Stack:

```
(gdb) bt
#0 0x00007fcc302b7ad6 in boost::asio::detail::epoll_reactor::start_op (this=0x309a6e0, op_type=op_type@entry=1, descriptor=1352, descriptor_data=@0x583560e88: 0x0,
op=op@entry=0x6cdf6e900, is_continuation=is_continuation@entry=true, allow_speculative=allow_speculative@entry=true)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/impl/epoll_reactor.ipp:219
#1 0x00007fcc302bca55 in start_op (noop=false, is_non_blocking=true, is_continuation=true, op=0x6cdf6e900, op_type=1, impl=..., this=0x309c418)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/impl/reactive_socket_service_base.ipp:214
#2 async_send >, boost::asio::detail::write_op, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> > (flags=0,
handler=..., buffers=..., impl=..., this=0x309c418) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/reactive_socket_service_base.hpp:216
#3 async_send >, boost::asio::detail::write_op, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> > (flags=0,
handler=, buffers=..., impl=..., this=0x309c3f0)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/stream_socket_service.hpp:330
#4 async_write_some >, boost::asio::detail::write_op, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> > (
handler=, buffers=..., this=0x583560e80)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/basic_stream_socket.hpp:732
#5 boost::asio::detail::write_op >, std::vector >, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3>::operator()(const boost::system::error_code &, std::size_t, int) (this=0x7fcc185cf030, ec=..., bytes_transferred=, start=)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/impl/write.hpp:181
#6 0x00007fcc302bd39d in operator() (this=0x7fcc185cf030) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/bind_handler.hpp:127
#7 asio_handler_invoke, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3>, boost::system::error_code, long unsigned int> > (function=...)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/handler_invoke_hook.hpp:69
#8 invoke, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3>, boost::system::error_code, long unsigned int>, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> (context=..., function=...) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#9 asio_handler_invoke, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3>, boost::system::error_code, long unsigned int>, boost::asio::basic_stream_socket, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> (
this_handler=, function=...) at /home/wutao1/boost_1_58_0/output/include/boost/asio/impl/write.hpp:565
#10 invoke, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3>, boost::system::error_code, long unsigned int>, boost::asio::detail::write_op, std::vector, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> > (
context=..., function=...) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#11 boost::asio::detail::reactive_socket_send_op > >, boost::asio::detail::write_op >, std::vector >, boost::asio::detail::transfer_all_t, dsn::tools::asio_rpc_session::send(uint64_t)::__lambda3> >::do_complete(boost::asio::detail::io_service_impl *, boost::asio::detail::operation *, const boost::system::error_code &, std::size_t) (owner=0x30b6620, base=)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/reactive_socket_send_op.hpp:107
#12 0x000000000074fec9 in complete (bytes_transferred=, ec=..., owner=..., this=)
at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/task_io_service_operation.hpp:38
---Type to continue, or q to quit---
#13 do_run_one (ec=..., this_thread=..., lock=..., this=0x30b6620) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/impl/task_io_service.ipp:372
#14 boost::asio::detail::task_io_service::run (this=0x30b6620, ec=...) at /home/wutao1/boost_1_58_0/output/include/boost/asio/detail/impl/task_io_service.ipp:149
#15 0x00007fcc302b2cc6 in run (this=, ec=...) at /home/wutao1/boost_1_58_0/output/include/boost/asio/impl/io_service.ipp:66
#16 operator() (__closure=0x30958b0) at /home/wutao1/pegasus-release/rdsn/src/core/tools/common/asio_net_provider.cpp:73
#17 _M_invoke<> (this=0x30958b0) at /home/wutao1/app/include/c++/4.8.2/functional:1732
#18 operator() (this=0x30958b0) at /home/wutao1/app/include/c++/4.8.2/functional:1720
#19 std::thread::_Impl >::_M_run(void) (this=0x3095898)
at /home/wutao1/app/include/c++/4.8.2/thread:115
#20 0x00007fcc2d035600 in std::(anonymous namespace)::execute_native_thread_routine (__p=)
at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#21 0x00007fcc2dca8dc5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fcc2c79f73d in clone () from /lib64/libc.so.6
```

4. What version of Pegasus are you using?

Pegasus-server 1.11.5

Contributor guide

Open the contributing guide

Research direction

Start with rdsn/src/core/tools/common/asio_net_provider.cpp at line 73 and trace dsn::tools::asio_rpc_session::send(uint64_t) into Boost.Asio's epoll_reactor::start_op in epoll_reactor.ipp. Use the supplied GDB backtrace and the normal bootstrap scenario to investigate the crash. Done means identifying the cause of the coredump and verifying that the scenario no longer crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.