apache / apache/incubator-pegasus
Replica server crashed when handle multiget request
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
1. What did you do?
If possible, provide a recipe for reproducing the error.
When rolling update a cluster from 2.0-write-optim to 2.1.1 we see one of replica server crashed.
The crashed server is `task 19`, that means we have already rolling update 19 servers successfully, but before rolling this server, it crashed.
2. What did you expect to see?
no crash.
3. What did you see instead?
Coredump stack:
```
(gdb) bt
#0 0x00000000b16a6700 in ?? ()
#1 0x000000000059b44f in rocksdb::DBImpl::MultiGet (this=0x1a641b400, read_options=..., column_family=..., keys=..., values=0x7f77a7a068a0)
at /home/wutao1/pegasus-release/rocksdb/db/db_impl/db_impl.cc:1742
#2 0x00000000005a79e1 in rocksdb::DB::MultiGet (this=, options=..., keys=..., values=0x7f77a7a068a0) at /home/wutao1/pegasus-release/rocksdb/include/rocksdb/db.h:450
#3 0x0000000000520136 in pegasus::server::pegasus_server_impl::on_multi_get (this=0x5bead00, request=..., reply=...)
at /home/wutao1/pegasus-release/src/server/pegasus_server_impl.cpp:611
#4 0x000000000050030f in bool dsn::replication::storage_serverlet::register_async_rpc_handler(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::apps::multi_get_request const&, dsn::rpc_replier&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}::operator()(dsn::apps::rrdb_service*, dsn::message_ex*) const () at /home/wutao1/pegasus-release/DSN_ROOT/include/dsn/dist/replication/storage_serverlet.h:29
#5 0x0000000000525b0c in operator() (__args#1=0x14cc4ea018, __args#0=0x5bead00, this=) at /home/wutao1/app/include/c++/4.8.2/functional:2464
#6 handle_request (request=0x14cc4ea018, this=0x5bead00) at /home/wutao1/pegasus-release/DSN_ROOT/include/dsn/dist/replication/storage_serverlet.h:80
#7 dsn::apps::rrdb_service::on_request (this=0x5bead00, request=0x14cc4ea018) at /home/wutao1/pegasus-release/src/include/rrdb/rrdb.server.h:17
#8 0x00007f77e24d2f82 in dsn::replication::replica::on_client_read (this=0xb56fce680, request=request@entry=0x14cc4ea018)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica.cpp:186
#9 0x00007f77e2544a9f in dsn::replication::replica_stub::on_client_read (this=0x3554600, id=..., request=0x14cc4ea018)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_stub.cpp:807
#10 0x00007f77e266c219 in dsn::task::exec_internal (this=this@entry=0x14cc4ea1b0) at /home/wutao1/pegasus-release/rdsn/src/core/core/task.cpp:180
#11 0x00007f77e268046d in dsn::task_worker::loop (this=0x3a00c60) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:211
#12 0x00007f77e2680639 in dsn::task_worker::run_internal (this=0x3a00c60) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:191
#13 0x00007f77def4e600 in std::(anonymous namespace)::execute_native_thread_routine (__p=)
at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#14 0x00007f77dfa60dc5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f77de6b873d in clone () from /lib64/libc.so.6
```
4. What version of Pegasus are you using?
pegasus-server-2.0-write-optim-984936d-glibc2.12-release
Contributor guide
Research direction
Start at src/server/pegasus_server_impl.cpp:611, following the MultiGet call through rocksdb/db/db_impl/db_impl.cc:1742 and the replica request path in rdsn/src/dist/replication/lib/replica.cpp:186. Reproduce the rolling update with a multiget request if possible; done means the replica server no longer crashes and the request completes safely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100