CorruptionError on compaction if cache is under pressure
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
### Expected behavior
Slowdown compaction/fail compaction
### Actual behavior
CorruptionError and in worse case invalid data in result files
### Steps to reproduce the behavior
Hight rate inserts with a very small cache
### Broken code sequence
```
#0 rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::GetContext*, rocksdb::Status) () at ../third-party/rocksdb-5.13.2/include/rocksdb/status.h:185
#1 0x0000555555de3ba2 in rocksdb::BlockBasedTableIterator::InitDataBlock() () at ../third-party/rocksdb-5.13.2/include/rocksdb/status.h:279
#2 0x0000555555de41f8 in rocksdb::BlockBasedTableIterator::SeekToFirst (this=0x615000070600) at ../third-party/rocksdb-5.13.2/table/block_based_table_reader.cc:1852
#3 0x0000555555dff720 in rocksdb::IteratorWrapper::SeekToFirst (this=0x613000012840) at ../third-party/rocksdb-5.13.2/table/iterator_wrapper.h:69
#4 rocksdb::MergingIterator::SeekToFirst (this=0x613000012800) at ../third-party/rocksdb-5.13.2/table/merging_iterator.cc:85
#5 0x0000555555e6404d in rocksdb::CompactionJob::ProcessKeyValueCompaction(rocksdb::CompactionJob::SubcompactionState*) () at ../third-party/rocksdb-5.13.2/db/compaction_job.cc:764
```
Lets take a closer look at ``BlockBasedTable::NewDataBlockIterator``:
```
//....lines skipped
if (s.ok()) {
// BUG: First part of the problem is here: iterator stored block.data_ pointer value internally, so
// no matter what is going on later it HAS a copy of the pointer already
iter = block.value->NewIterator(&rep->internal_comparator, iter, true,
rep->ioptions.statistics);
if (block.cache_handle != nullptr) {
iter->RegisterCleanup(&ReleaseCachedEntry, block_cache,
block.cache_handle);
} else {
if (!ro.fill_cache && rep->cache_key_prefix_size != 0) {
// insert a dummy record to block cache to track the memory usage
Cache::Handle* cache_handle;
// ... many irrelevant lines
s = block_cache->Insert(unique_key, nullptr, block.value->usable_size(), nullptr, &cache_handle);
if (s.ok()) {
if (cache_handle != nullptr) {
iter->RegisterCleanup(&ForceReleaseCachedEntry, block_cache,
cache_handle);
}
} else {
// BUG: Second part of the problem. Block memory has been deallocated, iter however still has
// deleted address inside its private data_ pointer. Its not null and iter WILL use it
delete block.value;
block.value = nullptr;
iter->data_ = nullptr;
}
```
In case if dummy node can't be placed in the cache, compaction will get BlockIter that is trying to interpret some random memory address as block content. It leads to some suspicious, random``CorruptionError`` that can't be reproduced on restart and, in worse case, could leads to invalid data in result file, if address happens to points to a valid block of some other SST
Contributor guide
Assessment
This issue has not been assessed yet.