cockroachdb / cockroachdb/pebble

db: possible memtable large value optimizations

Open
#2,747 0 comments 0 reactions 0 assignees View on GitHub
A-storage C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Thanks to the value blocks work, we now have `base.LazyValue` threaded throughout the iterator stack. I suspect there's an opportunity to flush less frequently, which can help reduce write-amp (especially in the context of CockroachDB, where delaying flushes makes raft log truncation more likely to drop keys before they're flushed).

A couple high-level thoughts:
1. If a value is large and compresses easily, a batch could compress it before entering the commit pipeline. When inserting into the memtable skiplist, it could set a flag on the node indicating the value is compressed and copy the smaller, compressed value into the arena. The value would need to be decompressed during flush, and during reads but only when that individual KV is returned from the pebble.Iterator.
2. Similar to generalized blob storage #112, the memtable entry could encode the position of the large value in the WAL. In the rare case that the blob must be read before it's flushed, reads would suffer I/O. Additionally, at flush time the value would need to be read back these values from the WAL. This is complicated by the WAL's / record package's framing which may split a value across frames/blocks. In these cases the memtable value could encode a list of (offset, length) tuples.

Jira issue: PEBBLE-63

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.