facebook / facebook/rocksdb

[BUG] RocksDB panics on rocksdb::InternalKeyComparator::Compare

Open
#3,129 0 comments 0 reactions 0 assignees View on GitHub
bug up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

This bug is a cross-post from [rust-rocksdb issue #148](https://github.com/spacejam/rust-rocksdb/issues/148)

I was working on a project that relies on rust-rocksdb, and got a core dump while the program was compacting the internal rocksdb.

Below is a core dump, and some basic analysis of the bug. I'm still working on figuring this bug out, but the bug seems to be coming from this C++ library.

Line #6 in the core dump is the relevant call before the crash.

Core Dump:
```
#0 0x00007f189355d8a0 in raise () from /usr/lib/libc.so.6
#1 0x00007f189355ef09 in abort () from /usr/lib/libc.so.6
#2 0x00007f1894ab7d77 in __gnu_cxx::__verbose_terminate_handler () at /build/gcc/src/gcc/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x00007f1894ab58e6 in __cxxabiv1::__terminate (handler=) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x00007f1894ab5933 in std::terminate () at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x00007f1894ab6741 in __cxxabiv1::__cxa_pure_virtual () at /build/gcc/src/gcc/libstdc++-v3/libsupc++/pure.cc:50
#6 0x000055c0a7d7dfe8 in rocksdb::InternalKeyComparator::Compare (this=0x7f188a441840, akey=..., bkey=...) at rocksdb/db/dbformat.cc:73
#7 0x000055c0a7cab41a in rocksdb::MinIteratorComparator::operator() (this=0x7f1885216868, a=0x7f1885216820, b=0x7f18852167e0) at rocksdb/table/iter_heap.h:36
#8 0x000055c0a7caded1 in rocksdb::BinaryHeap::downheap (this=0x7f1885216868, index=0) at rocksdb/util/heap.h:123
#9 0x000055c0a7cad240 in rocksdb::BinaryHeap::replace_top (this=0x7f1885216868,
value=@0x7f1885216858: 0x7f18852167c0) at rocksdb/util/heap.h:63
#10 0x000055c0a7cac20a in rocksdb::MergingIterator::Next (this=0x7f1885216780) at rocksdb/table/merger.cc:164
#11 0x000055c0a7d55114 in rocksdb::CompactionIterator::Next (this=0x7f1885852000) at rocksdb/db/compaction_iterator.cc:103
#12 0x000055c0a7d5be33 in rocksdb::CompactionJob::ProcessKeyValueCompaction (this=0x7f18873fc690, sub_compact=0x7f1885239700) at rocksdb/db/compaction_job.cc:845
#13 0x000055c0a7d5a0fe in rocksdb::CompactionJob::Run (this=0x7f18873fc690) at rocksdb/db/compaction_job.cc:530
#14 0x000055c0a7b7bfff in rocksdb::DBImpl::BackgroundCompaction (this=0x7f188a4cf000, made_progress=0x7f18873fcd3f, job_context=0x7f18873fcda0, log_buffer=0x7f18873fcf90,
arg=0x0) at rocksdb/db/db_impl.cc:3502
#15 0x000055c0a7b7a6c1 in rocksdb::DBImpl::BackgroundCallCompaction (this=0x7f188a4cf000, arg=0x0) at rocksdb/db/db_impl.cc:3196
#16 0x000055c0a7b79957 in rocksdb::DBImpl::BGWorkCompaction (arg=0x7f1885414090) at rocksdb/db/db_impl.cc:3002
#17 0x000055c0a7da553d in rocksdb::ThreadPool::BGThread (this=0x7f1892d28dc0, thread_id=1) at rocksdb/util/threadpool.cc:230
#18 0x000055c0a7da55f2 in rocksdb::BGThreadWrapper (arg=0x7f18922cf360) at rocksdb/util/threadpool.cc:254
#19 0x00007f1893afe08a in start_thread () from /usr/lib/libpthread.so.0
#20 0x00007f189361f24f in clone () from /usr/lib/libc.so.6
```

Key line from the core dump excerpt above:
- #6 0x000055c0a7d7dfe8 in rocksdb::InternalKeyComparator::Compare (this=0x7f188a441840, akey=..., bkey=...) at rocksdb/db/dbformat.cc:73

The line numbers changed in the github repo, but here's the relevant function definition for rocksdb::InternalKeyComparator::Compare from the C++ library:

```
int InternalKeyComparator::Compare(const Slice& akey, const Slice& bkey) const {
// Order by:
// increasing user key (according to user-supplied comparator)
// decreasing sequence number
// decreasing type (though sequence# should be enough to disambiguate)
int r = user_comparator_->Compare(ExtractUserKey(akey), ExtractUserKey(bkey));
PERF_COUNTER_ADD(user_key_comparison_count, 1);
if (r == 0) {
const uint64_t anum = DecodeFixed64(akey.data() + akey.size() - 8);
const uint64_t bnum = DecodeFixed64(bkey.data() + bkey.size() - 8);
if (anum > bnum) {
r = -1;
} else if (anum < bnum) {
r = +1;
}
}
return r;
}
```

and the relevant github link [facebook/rocksdb/db/dbformat.cc:109](https://github.com/facebook/rocksdb/blob/master/db/dbformat.cc#L109) (w/ adjusted line #).

PERF_COUNTER_ADD looks like may be the culprit, defined here:
```
monitoring/perf_context_imp.h

30: #define PERF_COUNTER_ADD(metric, value)
...
...
55: #define PERF_COUNTER_ADD(metric, value) \
56: if (perf_level >= PerfLevel::kEnableCount) { \
57: perf_context.metric += value; \
58: }
59:
60: #endif
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.