apache / apache/incubator-pegasus
coredump when replay private log
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Report
Time: 2019/12/11
Version: 1.11.6
### Coredump
```
(gdb) bt
#0 0x00007f217c4ab1d7 in raise () from /lib64/libc.so.6
#1 0x00007f217c4ac8c8 in abort () from /lib64/libc.so.6
#2 0x00007f217ffea9fe in dsn_coredump () at /home/wutao1/pegasus-release/rdsn/src/core/core/service_api_c.cpp:76
#3 0x00007f217fecf589 in dsn::replication::prepare_list::commit (this=this@entry=0x7f213933f010, d=16635787, ct=ct@entry=dsn::replication::COMMIT_TO_DECREE_HARD)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:141
#4 0x00007f217fecfa12 in dsn::replication::prepare_list::prepare (this=0x7f213933f010, mu=..., status=status@entry=dsn::replication::partition_status::PS_SECONDARY)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:80
#5 0x00007f217ff136f9 in operator() (log_length=, mu=..., __closure=0x48418c258) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_learn.cpp:1415
#6 std::_Function_handler&), dsn::replication::replica::apply_learned_state_from_private_log(dsn::replication::learn_state&)::__lambda24>::_M_invoke(const std::_Any_data &, int, dsn::ref_ptr &) (__functor=..., __args#0=, __args#1=...) at /home/wutao1/app/include/c++/4.8.2/functional:2057
#7 0x00007f217fec4aa1 in dsn::replication::mutation_log::replay(dsn::ref_ptr, std::function&)>, long&) (log=..., callback=...,
end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:885
#8 0x00007f217fec4e9f in dsn::replication::mutation_log::replay(std::map, std::less, std::allocator > > >&, std::function&)>, long&) (logs=..., callback=..., end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:967
#9 0x00007f217fec7415 in dsn::replication::mutation_log::replay(std::vector >&, std::function&)>, long&) (log_files=..., callback=...,
end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:927
#10 0x00007f217ff13ea9 in dsn::replication::replica::apply_learned_state_from_private_log (this=this@entry=0xa37885600, state=...) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_learn.cpp:1418
#11 0x00007f217fef2ae6 in dsn::replication::replica::catch_up_with_private_logs (this=0xa37885600, s=dsn::replication::partition_status::PS_POTENTIAL_SECONDARY)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_chkpt.cpp:335
#12 0x00007f217fffccd9 in dsn::task::exec_internal (this=this@entry=0x11c99c59d4) at /home/wutao1/pegasus-release/rdsn/src/core/core/task.cpp:180
#13 0x00007f2180010a6d in dsn::task_worker::loop (this=0x2a7b3f0) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:211
#14 0x00007f2180010c39 in dsn::task_worker::run_internal (this=0x2a7b3f0) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:191
#15 0x00007f217ce03600 in std::(anonymous namespace)::execute_native_thread_routine (__p=) at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#16 0x00007f217da76dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f217c56d73d in clone () from /lib64/libc.so.6
```
```
(gdb) f 3
#3 0x00007f217fecf589 in dsn::replication::prepare_list::commit (this=this@entry=0x7f213933f010, d=16635787, ct=ct@entry=dsn::replication::COMMIT_TO_DECREE_HARD)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:141
141 /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp: No such file or directory.
(gdb) p mu
$1 = {_obj = 0x0}
```
It seems that mutation whose decree is 16635787 is nullptr and is missing in prepare list.
Contributor guide
Research direction
Start with dist/replication/lib/prepare_list.cpp at commit line 141 and trace the call from prepare. Then inspect replica_learn.cpp around apply_learned_state_from_private_log and mutation_log.cpp replay at the reported lines, using the provided private-log replay scenario. Done means the missing mutation case is understood and replay no longer reaches the reported coredump.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100