facebook / facebook/rocksdb

SIGSEGV in rocksdb::MergingIterator::Next() when CompactionJob: Version 7.1.1

Open
#10,755 3 comments 0 reactions 0 assignees View on GitHub
crash java up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

**### Expected behavior:**
RocksDB works well during YCSB load phase

**### Actual behavior:**
Each time RocksDB worked well at the beginning but then run into SIGSEGV after 1 hour or 3 hours during YCSB load phase.
And it always happened at rocksdb::MergingIterator::Next() when doing BG Compaction.
Below is the frame when error occurs:
Stack: [0x00007f2a7abe7000,0x00007f2a7b3e6000], sp=0x00007f2a7b3e3300, free space=8176k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni609321496660644933.so+0x58fd9a] rocksdb::Customizable::AreEquivalent(rocksdb::ConfigOptions const&, rocksdb::Configurable const*, std::string*) const+0x3a
C [librocksdbjni609321496660644933.so+0x644d45] rocksdb::MergingIterator::Next()+0x2d5
C [librocksdbjni609321496660644933.so+0x323dee] rocksdb::CompactionIterator::Next()+0x1be
C [librocksdbjni609321496660644933.so+0x332b36] rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*)+0xca6
C [librocksdbjni609321496660644933.so+0x334213] rocksdb::CompactionJob::Run()+0x273
C [librocksdbjni609321496660644933.so+0x3ac638] rocksdb::DBImpl::BackgroundCompaction(bool*, rocksdb::JobContext*, rocksdb::LogBuffer*, rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xf88
C [librocksdbjni609321496660644933.so+0x3ad847] rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority)+0xd7
C [librocksdbjni609321496660644933.so+0x3add8e] rocksdb::DBImpl::BGWorkCompaction(void*)+0x4e
C [librocksdbjni609321496660644933.so+0x6c1878] rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long)+0x228
C [librocksdbjni609321496660644933.so+0x6c1a42] rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*)+0x62

For more information, please see the attached two files about the backtrace dump.

**### Steps to reproduce the behavior**

1. compile ycsb using rocksdb version 7.1.1
2. run cmd under YCSB directory
`./bin/ycsb load rocksdb -s -P workloads/workloada -P workloads/large.dat -p columnfamily=default`

large.dat:
` recordcount=85000000
threadcount=2
operationcount=85000000
measurementtype=timeseries
timeseries.granularity=1000`

[hs_err_pid13571.txt](https://github.com/facebook/rocksdb/files/9669854/hs_err_pid13571.txt)
[hs_err_pid15146.txt](https://github.com/facebook/rocksdb/files/9669855/hs_err_pid15146.txt)

Contributor guide

Open the contributing guide

Research direction

Start with the attached hs_err files and the stack entry at rocksdb::MergingIterator::Next(), then trace the compaction path through CompactionIterator and CompactionJob. Reproduce with the listed YCSB load command and workload settings. Done means identifying the cause of the SIGSEGV and validating a fix under the same workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.