drop deletion markers in compaction when they don't cover any data
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
When compacting to bottom level (the level in the LSM containing the oldest data) we currently output deletion markers ("tombstones") if they're newer than any snapshot. That's because the key covered by a tombstone may still be visible to a snapshot, so we can't drop either it or its tombstone yet.
One time, this caused an issue for MyRocks where there were a huge number of tombstones output by compaction even though the keys they covered had already been dropped (they were either non-existent initially or no longer visible to any snapshots). We should make our check more precise and only write tombstones out if they also actually cover some real key, not just check if it's theoretically possible for them to cover a key. This will require doing some lookahead on the compaction iterator to check what a tombstone covers before deciding whether to output it or not.
Contributor guide
Assessment
This issue has not been assessed yet.