apache / apache/incubator-pegasus

Bug: coredump in function last_prepared_decree()

Open
#516 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report
What version of Pegasus are you using?
`Pegasus Server 1.12.1 (694cbd544436f03d34bfbfcbca0cd9b8f397197e)`

Coredump Stack
```
(gdb) bt
#0 0x00007f571187e1d7 in raise () from /lib64/libc.so.6
#1 0x00007f571187f8c8 in abort () from /lib64/libc.so.6
#2 0x00007f5711877146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f57118771f2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f57155dc98f in dsn::ref_counter::release_ref (this=)
at /home/wutao1/pegasus-release/rdsn/include/dsn/utility/autoref_ptr.h:71
#5 0x00007f57156bef1c in release_ref (this=)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica.cpp:338
#6 ~ref_ptr (this=0x7f56cc5e2fc0, __in_chrg=)
at /home/wutao1/pegasus-release/rdsn/include/dsn/utility/autoref_ptr.h:142
#7 dsn::replication::replica::last_prepared_decree (this=this@entry=0x14187200)
at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica.cpp:336
#8 0x00007f57156ff742 in dsn::replication::replica::on_copy_remote_state_completed(dsn::error_code, unsigned long, unsigned long, dsn::replication::learn_request&&, dsn::replication::learn_response&&) (this=0x14187200, err=...,
size=0, copy_start_time=152,
req=,
resp=) at /home/wutao1/pegasus-release/rdsn/src/dist/replication/lib/replica_learn.cpp:913
#9 0x00007f57158208a9 in dsn::task::exec_internal (this=this@entry=0x789b277af)
at /home/wutao1/pegasus-release/rdsn/src/core/core/task.cpp:180
---Type to continue, or q to quit---
#10 0x00007f5715834a6d in dsn::task_worker::loop (this=0x1da73f0)
at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:211
#11 0x00007f5715834c39 in dsn::task_worker::run_internal (this=0x1da73f0)
at /home/wutao1/pegasus-release/rdsn/src/core/core/task_worker.cpp:191
#12 0x00007f57121d6600 in std::(anonymous namespace)::execute_native_thread_routine (__p=)
at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#13 0x00007f5712d3bdc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f571194073d in clone () from /lib64/libc.so.6
```

Information from frame 7:
replica gpid is 10.55
mutation is 10.55.34.194880435
start decree is 194880434

Information from frame 6:
autoref_ptr is dsn::ref_ptr * const
mutation is 10.55.34.194880435

Corresponding code:
https://github.com/XiaoMi/rdsn/blob/4b14d3adeec056736d621645c15d2ef0e495b97e/src/dist/replication/lib/replica.cpp#L325-L338

https://github.com/XiaoMi/rdsn/blob/4b14d3adeec056736d621645c15d2ef0e495b97e/include/dsn/utility/autoref_ptr.h#L68-L71

This bug seems to caused by trying to release reference the mutation structure who has already been released.

Contributor guide

Open the contributing guide

Research direction

Start with replica.cpp lines 325-338 and autoref_ptr.h lines 68-71, then trace the call from replica_learn.cpp line 913. Use the reported gpid, mutation, and start decree to understand the reference-release path. Done means the reported last_prepared_decree() path no longer aborts while releasing the mutation reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.