Filters are not cached when both cache_index_and_filter_blocks and allow_mmap_reads are set
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
After attempting to upgrade from an older version of RocksDB to a newer one that includes #5298 and #5504, I started seeing a very significant slowdown in a point-lookup heavy workload. Profiling indicated that filter blocks are repeatedly being read and checksummed (`rocksdb::FilterBlockReaderCommon::GetOrReadFilterBlock` invoking `rocksdb::BlockFetcher::ReadBlockContents` and `rocksdb::VerifyBlockChecksum` constantly, when they were not being invoked before).
The following test program demonstrates the issue. It performs a 100 Puts and then performs 100 Gets with a perf context. Note `cache_index_and_filter_blocks` and `allow_mmap_reads` are set. When executed with RocksDB 6.4 and later, 100 filter block reads occur:
```c++
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
static rocksdb::Status openDB(rocksdb::Options const& opts, std::string const& name, std::unique_ptr& db)
{
rocksdb::DB* pdb{};
rocksdb::Status s = rocksdb::DB::Open(opts, name, &pdb);
if (s.ok()) {
db.reset(pdb);
}
return s;
}
static void check_status(rocksdb::Status s)
{
if (!s.ok()) {
std::cerr << s.ToString() << '\n';
abort();
}
}
int main()
{
rocksdb::Options opts;
opts.create_if_missing = true;
opts.statistics = rocksdb::CreateDBStatistics();
opts.compression = rocksdb::kLZ4Compression;
opts.allow_mmap_reads = true;
rocksdb::BlockBasedTableOptions bbto;
bbto.cache_index_and_filter_blocks = true;
bbto.block_cache = rocksdb::NewLRUCache(32 * 1024 * 1024);
bbto.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));
opts.table_factory.reset(rocksdb::NewBlockBasedTableFactory(bbto));
rocksdb::DestroyDB("rdb_test", opts);
std::unique_ptr db;
rocksdb::Status s = openDB(opts, "rdb_test", db);
check_status(s);
constexpr int keyCount = 100;
rocksdb::WriteBatch wb;
for (int i = 0; i < keyCount; i++) {
auto kv = std::to_string(i);
s = wb.Put(kv, kv);
check_status(s);
}
rocksdb::WriteOptions wo;
s = db->Write(wo, &wb);
check_status(s);
rocksdb::FlushOptions fo;
s = db->Flush(fo);
check_status(s);
rocksdb::ReadOptions ro;
rocksdb::SetPerfLevel(rocksdb::PerfLevel::kEnableCount);
auto pc = rocksdb::get_perf_context();
pc->EnablePerLevelPerfContext();
pc->Reset();
std::string vd;
for (int i = 0; i < keyCount; i++) {
auto kv = std::to_string(i);
s = db->Get(ro, kv, &vd);
if (!s.ok() && !s.IsNotFound()) {
abort();
}
}
std::cout << "Filter block read count: " << pc->filter_block_read_count << '\n';
pc->Reset();
s = db->Close();
check_status(s);
return 0;
}
```
With RocksDB 6.4 and higher the output is:
```
Filter block read count: 100
```
With RocksDB 6.3.6 and earlier, or with RocksDB 6.4 and higher with either `allow_mmap_reads = false` or `cache_index_and_filter_blocks = false`:
```
Filter block read count: 0
```
### Expected behavior
Filter blocks should be cached and not read and checksummed repeatedly when `cache_index_and_filter_blocks` and `allow_mmap_reads` are both set.
### Actual behavior
If both `cache_index_and_filter_blocks` and `allow_mmap_reads` are set, filter blocks are read repeatedly.
### Steps to reproduce the behavior
See above program.
@ltamasi - FYI.
Contributor guide
Assessment
This issue has not been assessed yet.