apache / apache/incubator-pegasus
Bug(Learn):bakcup or dup a table with per disk throttling on a backup-duplication cluster , some nodes coredump
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
1. What did you do?
Firstly , I do all my work in a bakcup-duplication cluster (relatively,mater-duplication cluster send message to it.)
- When I config `max_copy_rate_megabytes_per_disk` , Whether I do backup or duplication for an app , some nodes on bakcup-cluster will coredump.And I discovered a phenomenon that they all coredump **when secondary replica try to learn from primary replica**.
- Without `max_copy_rate_megabytes_per_disk` , backup action or duplication action will not casue any coredump.
2.coredump file:
```
#0 0x00007fa027148c30 in dsn::replication::replica_duplicator_manager::update_duplication_map (this=0x0, new_dup_map=...)
at /home/work/pegasus/src/rdsn/src/replica/duplication/replica_duplicator_manager.h:57
57 /home/work/pegasus/src/rdsn/src/replica/duplication/replica_duplicator_manager.h: No such file or directory.
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-15.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-67.el7_7.1.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) #0 0x00007fa027148c30 in dsn::replication::replica_duplicator_manager::update_duplication_map (this=0x0, new_dup_map=...)
at /home/work/pegasus/src/rdsn/src/replica/duplication/replica_duplicator_manager.h:57
#1 0x00007fa027146635 in dsn::replication::duplication_sync_timer::update_duplication_map (this=this@entry=0x407c5f00, dup_map=...)
at /home/work/pegasus/src/rdsn/src/replica/duplication/duplication_sync_timer.cpp:111
#2 0x00007fa02714673b in dsn::replication::duplication_sync_timer::on_duplication_sync_reply (this=this@entry=0x407c5f00, err=..., resp=...)
at /home/work/pegasus/src/rdsn/src/replica/duplication/duplication_sync_timer.cpp:90
#3 0x00007fa027146825 in operator() (err=..., __closure=0x8f0239e0)
at /home/work/pegasus/src/rdsn/src/replica/duplication/duplication_sync_timer.cpp:77
#4 operator() (req=, resp=, err=...,
__closure=0x8f0239e0)
at /home/work/pegasus/src/rdsn/include/dsn/cpp/rpc_holder.h:161
#5 std::_Function_handler::call(const dsn::rpc_address&, dsn::task_tracker*, TCallback&&, int) [with TCallback = dsn::replication::duplication_sync_timer::run()::; TRequest = dsn::replication::duplication_sync_request; TResponse = dsn::replication::duplication_sync_response; dsn::task_ptr = dsn::ref_ptr]:: >::_M_invoke(const std::_Any_data &, dsn::error_code &, dsn::message_ex *&, dsn::message_ex *&) (__functor=..., __args#0=...,
__args#1=, __args#2=)
at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:316
#6 0x00007fa0272c06d7 in operator() (__args#2=,
__args#1=, __args#0=..., this=)
at /opt/rh/devtoolset-7/root/usr/include/c++/7/bits/std_function.h:706
#7 dsn::rpc_response_task::exec (this=)
at /home/work/pegasus/src/rdsn/include/dsn/tool-api/task.h:478
#8 0x00007fa0272c1431 in dsn::task::exec_internal (
this=this@entry=0x8d6b2b00)
at /home/work/pegasus/src/rdsn/src/runtime/task/task.cpp:176
#9 0x00007fa0272d6ae2 in dsn::task_worker::loop (this=0x2e66e70)
at /home/work/pegasus/src/rdsn/src/runtime/task/task_worker.cpp:224
#10 0x00007fa0272d6c60 in dsn::task_worker::run_internal (this=0x2e66e70)
at /home/work/pegasus/src/rdsn/src/runtime/task/task_worker.cpp:204
#11 0x00007fa025f54a1f in execute_native_thread_routine ()
from /home/work/app/pegasus/azpnsrv-storecom/replica/package/bin/libdsn_utils.so
#12 0x00007fa023d5fdc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fa02225e73d in clone () from /lib64/libc.so.6
(gdb) quit
```
3. What version of Pegasus are you using?
v2.4
Contributor guide
Research direction
Start with rdsn/src/replica/duplication/replica_duplicator_manager.h at update_duplication_map and follow the calls from duplication_sync_timer.cpp lines 77, 90, and 111. Reproduce backup or duplication with max_copy_rate_megabytes_per_disk enabled on Pegasus v2.4, then verify secondary-replica synchronization no longer reaches a null manager or causes node coredumps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100