apache / apache/incubator-pegasus

Assert from rocksdb posix_logger

Open
#365 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. What did you do?
If possible, provide a recipe for reproducing the error.

Normal rolling-update on an existing cluster. (c4tst-perfomance)

### 2. What did you expect to see?

No error.

### 3. What did you see instead?

pegasus_server: /home/wutao1/pegasus/rocksdb/env/posix_logger.h:151: virtual void rocksdb::PosixLogger::Logv(const char*, __va_list_tag*): Assertion `sz == write_size' failed.

```
(gdb) bt
#0 0x00007f7845b641d7 in raise () from /lib64/libc.so.6
#1 0x00007f7845b658c8 in abort () from /lib64/libc.so.6
#2 0x00007f7845b5d146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f7845b5d1f2 in __assert_fail () from /lib64/libc.so.6
#4 0x0000000000924572 in rocksdb::PosixLogger::Logv(char const*, __va_list_tag*) ()
#5 0x0000000000896cb3 in rocksdb::Log(rocksdb::InfoLogLevel, rocksdb::Logger*, char const*, ...) ()
#6 0x00000000008a35ac in rocksdb::ImmutableDBOptions::Dump(rocksdb::Logger*) const ()
#7 0x00000000007e6fd3 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::string const&) ()
#8 0x000000000081c0ef in rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector > const&, std::vector >*, rocksdb::DB**) ()
#9 0x000000000081d856 in rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**) ()
#10 0x0000000000701e98 in pegasus::server::pegasus_server_impl::start (this=0x57d0000, argc=, argv=)
at /home/wutao1/pegasus/src/server/pegasus_server_impl.cpp:1515
#11 0x00007f784964e08b in dsn::replication::replication_app_base::open (this=this@entry=0x57d0000)
at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replication_app_base.cpp:425
#12 0x00007f7849650968 in dsn::replication::replication_app_base::open_internal (this=0x57d0000, r=r@entry=0x2f6f700)
at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replication_app_base.cpp:346
#13 0x00007f784960e0af in dsn::replication::replica::init_app_and_prepare_list (this=this@entry=0x2f6f700, create_new=create_new@entry=false)
at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:211
#14 0x00007f784960f45a in dsn::replication::replica::initialize_on_load (this=this@entry=0x2f6f700) at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:121
#15 0x00007f784960f88a in dsn::replication::replica::load (stub=0x27de580, dir=0x2e185b8 "/home/work/ssd6/pegasus/c4tst-perfomance/replica/reps/60.33.pegasus")
at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_init.cpp:162
#16 0x00007f784962e801 in operator() (__closure=0x2cf8fe0) at /home/wutao1/pegasus/rdsn/src/dist/replication/lib/replica_stub.cpp:397
#17 std::_Function_handler::_M_invoke(const std::_Any_data &) (
__functor=...) at /home/wutao1/app/include/c++/4.8.2/functional:2071
#18 0x00007f784970b149 in dsn::task::exec_internal (this=this@entry=0x2e1ec60) at /home/wutao1/pegasus/rdsn/src/core/core/task.cpp:180
#19 0x00007f784971eedd in dsn::task_worker::loop (this=0x2c704d0) at /home/wutao1/pegasus/rdsn/src/core/core/task_worker.cpp:211
#20 0x00007f784971f0a9 in dsn::task_worker::run_internal (this=0x2c704d0) at /home/wutao1/pegasus/rdsn/src/core/core/task_worker.cpp:191
#21 0x00007f78464bc600 in std::(anonymous namespace)::execute_native_thread_routine (__p=)
at /home/qinzuoyan/git.xiaomi/pegasus/toolchain/objdir/../gcc-4.8.2/libstdc++-v3/src/c++11/thread.cc:84
#22 0x00007f784712fdc5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f7845c2673d in clone () from /lib64/libc.so.6
```

### 4. What version of Pegasus are you using?

pegasus (bdc029937bee99025c69b6a7f71ef4d31a1c5abb)
rocksdb (dc91f8a9567b65cea9cc7d1cc792ddd764141638)

The cause for this error may be the incompatible upgrade of rocksdb.

Contributor guide

Open the contributing guide

Research direction

Start at rocksdb/env/posix_logger.h:151 and follow the stack through ImmutableDBOptions::Dump and pegasus_server_impl.cpp:1515. Compare the listed Pegasus and RocksDB revisions, reproduce the assertion during a rolling update if possible, and establish a verified fix or compatibility requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.