cockroachdb / cockroachdb/pebble

db: separate garbage keys into a separate sstable block

Open
#5,040 2 comments 0 reactions 0 assignees View on GitHub
A-storage C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

In #4424 we consider separating the values of keys that appear to be garbage (i.e., there exists another key with the same prefix but a larger suffix). This has the advantage of improving cache locality during reads, and improving write amplification by allowing us to more eagerly separate values when we suspect they're garbage. We could also consider separating the garbage MVCC _keys_ themselves. This would further improve the cache efficiency for reads at current timestamps.

During sstable building, if a key has the same prefix as the predecessor, we would write it to a separate garbage block that we build alongside the current data block. We'd set a bit in a bitmap in the primary data block indicating that there exist keys with the same prefix in the garbage block. During a MVCCGet, the iterator would seek for the prefix, land on the most recent KV in the primary data block and avoid needing to load the garbage data block unless the iterator is Nexted (only happens when the iterator is reading at an older timestamp). During a forward MVCCScan, the iterator would use NextPrefix to step between live keys. Each call to NextPrefix can step to the next key in the primary data block without loading the garbage block. We'd need to take care to ensure we're not accidentally Next-ing the iterator, which would force a load of the garbage block.

The semantics for a reverse MVCC scan are a bit trickier and would require more thought.

----

**More complicated, hand-wavy extension**

EDIT: This is incomplete as described; if there are other versions of the key lower in the LSM, we need to know that we should still surface the MVCC tombstone.

When we you have a swath of data that's MVCC tombstoned, the above scheme still requires storing the tombstones in the primary data block. This is unfortunate, because a scan still needs to step through all of the MVCC tombstones to get to something that's live. We'd like to be able to separate even the most recent key when it's a tombstone (effectively 'garbage' from the perspective of reads). This is challenging because the tombstone may have been written by a transaction that was aborted, making the shadowed key live. A read may need to observe the shadowed key if the MVCC tombstone is deleted by a transaction abort's DEL sitting higher in the LSM.

An alternative is to be optimistic, leaving enough information in the LSM to know when our optimism was misguided.

During compactions, we separate MVCC tombstone keys into the garbage block as well. In the primary data block, we can mark a bit indicating that there exists garbage keys between the previous live key and the next live key. The primary data block's block header can also indicate the maximum timestamp of any key that caused us to consider something as garbage. We can call this the 'garbage threshold. For example with `k@5=v1`, `k@3=v2` the `@5` is the garbage threshold and is stored in the block header.

We also add two new key kinds `DELGC` and `DELABORT` used specifically for MVCC GC and transaction aborts respectively. The `DELGC` is permitted to be stored in the accompanying garbage block like a MVCC tombstone would. A `DELABORT` must always be kept in the live block.

During a MVCCGet, we first perform the SeekPrefixGE indicating that we should ignore any garbage blocks as long as the data block's garbage threshold is less than our read timestamp. If we observe a `DELABORT`, it's possible that some garbage data block contains a relevant key. We reissue the read. This time, if the live data block indicates there are garbage keys in the gap where the seek key falls, we read and merge in the garbage block.

During a forward MVCCScan, if a `DELABORT` rises to the top of the heap, any iterator for a level lower in the LSM than the `DELABORT` must be repositioned to any garbage for keys >= the `DELABORT`'s user key.

Reverse MVCCScans would require even more thought.

Jira issue: PEBBLE-1079

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the sstable-building and compaction paths, then trace how MVCCGet, forward MVCCScan, and reverse MVCCScan use their iterators. Work through the primary and garbage-block proposal, including the garbage threshold, DELGC, and DELABORT semantics. Done requires a resolved design that handles older LSM versions, aborted transactions, forward scans, and reverse scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.