apache / apache/incubator-pegasus

coredump when replay private log

Open
#441 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

### Bug Report
Time: 2019/12/11
Version: 1.11.6

### Coredump
```
(gdb) bt
#0 0x00007f217c4ab1d7 in raise () from /lib64/libc.so.6
#1 0x00007f217c4ac8c8 in abort () from /lib64/libc.so.6
#2 0x00007f217ffea9fe in dsn_coredump () at /home/wutao1/pegasus-release/rdsn/src/core/core/service_api_c.cpp:76
#3 0x00007f217fecf589 in dsn::replication::prepare_list::commit (this=this@entry=0x7f213933f010, d=16635787, ct=ct@entry=dsn::replication::COMMIT_TO_DECREE_HARD)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:141
#4 0x00007f217fecfa12 in dsn::replication::prepare_list::prepare (this=0x7f213933f010, mu=..., status=status@entry=dsn::replication::partition_status::PS_SECONDARY)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:80
#5 0x00007f217ff136f9 in operator() (log_length=, mu=..., __closure=0x48418c258) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_learn.cpp:1415
#6 std::_Function_handler&), dsn::replication::replica::apply_learned_state_from_private_log(dsn::replication::learn_state&)::__lambda24>::_M_invoke(const std::_Any_data &, int, dsn::ref_ptr &) (__functor=..., __args#0=, __args#1=...) at /home/wutao1/app/include/c++/4.8.2/functional:2057
#7 0x00007f217fec4aa1 in dsn::replication::mutation_log::replay(dsn::ref_ptr, std::function&)>, long&) (log=..., callback=...,
end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:885
#8 0x00007f217fec4e9f in dsn::replication::mutation_log::replay(std::map, std::less, std::allocator > > >&, std::function&)>, long&) (logs=..., callback=..., end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:967
#9 0x00007f217fec7415 in dsn::replication::mutation_log::replay(std::vector >&, std::function&)>, long&) (log_files=..., callback=...,
end_offset=@0x7f213933efb8: 3490466037) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/mutation_log.cpp:927
#10 0x00007f217ff13ea9 in dsn::replication::replica::apply_learned_state_from_private_log (this=this@entry=0xa37885600, state=...) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_learn.cpp:1418
#11 0x00007f217fef2ae6 in dsn::replication::replica::catch_up_with_private_logs (this=0xa37885600, s=dsn::replication::partition_status::PS_POTENTIAL_SECONDARY)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_chkpt.cpp:335
#12 0x00007f217fffccd9 in dsn::task::exec_internal (this=this@entry=0x11c99c59d4) at /home/wutao1/pegasus-release/rdsn/src/core/core/task.cpp:180
#13 0x00007f2180010a6d in dsn::task_worker::loop (this=0x2a7b3f0) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:211
#14 0x00007f2180010c39 in dsn::task_worker::run_internal (this=0x2a7b3f0) at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:191
#15 0x00007f217ce03600 in std::(anonymous namespace)::execute_native_thread_routine (__p=) at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#16 0x00007f217da76dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f217c56d73d in clone () from /lib64/libc.so.6
```
```
(gdb) f 3
#3 0x00007f217fecf589 in dsn::replication::prepare_list::commit (this=this@entry=0x7f213933f010, d=16635787, ct=ct@entry=dsn::replication::COMMIT_TO_DECREE_HARD)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp:141
141 /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/prepare_list.cpp: No such file or directory.
(gdb) p mu
$1 = {_obj = 0x0}
```
It seems that mutation whose decree is 16635787 is nullptr and is missing in prepare list.

Contributor guide

Open the contributing guide

Research direction

Start with dist/replication/lib/prepare_list.cpp at commit line 141 and trace the call from prepare. Then inspect replica_learn.cpp around apply_learned_state_from_private_log and mutation_log.cpp replay at the reported lines, using the provided private-log replay scenario. Done means the missing mutation case is understood and replay no longer reaches the reported coredump.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.