facebook / facebook/rocksdb

Improve lifetime management in RangeDelAggregator

Open
#4,807 1 comment 0 reactions 0 assignees View on GitHub
up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

Currently, any range tombstone iterator passed to `RangeDelAggregator` is retained for the aggregator's entire lifetime. For long range scans, this can have a significant memory cost. Ideally, we would free iterators when the top-level `DBIter` moves past its corresponding file's range (L1+ only), and re-create them if `DBIter` moves back to the file. This would manifest as a `RemoveTombstones` method that is called by `LevelIterator`.

The problem with this approach is that a `LevelIterator` advances files after the last point key in the file is read, even if there is a gap between this key and the start of the next file that is covered by a range tombstone. We do not want to preemptively destroy the iterator containing this range tombstone, as it might cover keys in lower levels.

One solution to this problem is to insert sentinel keys between files in a `LevelIterator` if a file boundary has type `kTypeRangeDeletion`. These keys can be skipped by `DBIter`, but since this key will be stored in the underlying heap of iterators, it will keep the tombstones in the file alive for as long as they need to be alive.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.