facebook / facebook/rocksdb

Filters are not cached when both cache_index_and_filter_blocks and allow_mmap_reads are set

Open
#9,367 6 comments 0 reactions 0 assignees View on GitHub
design discussion
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

After attempting to upgrade from an older version of RocksDB to a newer one that includes #5298 and #5504, I started seeing a very significant slowdown in a point-lookup heavy workload. Profiling indicated that filter blocks are repeatedly being read and checksummed (`rocksdb::FilterBlockReaderCommon::GetOrReadFilterBlock` invoking `rocksdb::BlockFetcher::ReadBlockContents` and `rocksdb::VerifyBlockChecksum` constantly, when they were not being invoked before).

The following test program demonstrates the issue. It performs a 100 Puts and then performs 100 Gets with a perf context. Note `cache_index_and_filter_blocks` and `allow_mmap_reads` are set. When executed with RocksDB 6.4 and later, 100 filter block reads occur:

```c++
#include
#include
#include
#include
#include
#include

#include
#include
#include
#include
#include
#include
#include

static rocksdb::Status openDB(rocksdb::Options const& opts, std::string const& name, std::unique_ptr& db)
{
rocksdb::DB* pdb{};
rocksdb::Status s = rocksdb::DB::Open(opts, name, &pdb);
if (s.ok()) {
db.reset(pdb);
}

return s;
}

static void check_status(rocksdb::Status s)
{
if (!s.ok()) {
std::cerr << s.ToString() << '\n';
abort();
}
}

int main()
{
rocksdb::Options opts;
opts.create_if_missing = true;
opts.statistics = rocksdb::CreateDBStatistics();
opts.compression = rocksdb::kLZ4Compression;
opts.allow_mmap_reads = true;

rocksdb::BlockBasedTableOptions bbto;
bbto.cache_index_and_filter_blocks = true;
bbto.block_cache = rocksdb::NewLRUCache(32 * 1024 * 1024);
bbto.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));

opts.table_factory.reset(rocksdb::NewBlockBasedTableFactory(bbto));

rocksdb::DestroyDB("rdb_test", opts);
std::unique_ptr db;
rocksdb::Status s = openDB(opts, "rdb_test", db);
check_status(s);

constexpr int keyCount = 100;

rocksdb::WriteBatch wb;
for (int i = 0; i < keyCount; i++) {
auto kv = std::to_string(i);
s = wb.Put(kv, kv);
check_status(s);
}

rocksdb::WriteOptions wo;
s = db->Write(wo, &wb);
check_status(s);

rocksdb::FlushOptions fo;
s = db->Flush(fo);
check_status(s);

rocksdb::ReadOptions ro;

rocksdb::SetPerfLevel(rocksdb::PerfLevel::kEnableCount);
auto pc = rocksdb::get_perf_context();
pc->EnablePerLevelPerfContext();
pc->Reset();

std::string vd;
for (int i = 0; i < keyCount; i++) {
auto kv = std::to_string(i);
s = db->Get(ro, kv, &vd);
if (!s.ok() && !s.IsNotFound()) {
abort();
}
}

std::cout << "Filter block read count: " << pc->filter_block_read_count << '\n';
pc->Reset();

s = db->Close();
check_status(s);

return 0;
}
```

With RocksDB 6.4 and higher the output is:
```
Filter block read count: 100
```
With RocksDB 6.3.6 and earlier, or with RocksDB 6.4 and higher with either `allow_mmap_reads = false` or `cache_index_and_filter_blocks = false`:
```
Filter block read count: 0
```

### Expected behavior
Filter blocks should be cached and not read and checksummed repeatedly when `cache_index_and_filter_blocks` and `allow_mmap_reads` are both set.
### Actual behavior
If both `cache_index_and_filter_blocks` and `allow_mmap_reads` are set, filter blocks are read repeatedly.
### Steps to reproduce the behavior
See above program.

@ltamasi - FYI.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.