cockroachdb / cockroachdb/pebble

db: consider memtable optimizations for duplicate keys

Open
#2,683 3 comments 0 reactions 0 assignees View on GitHub
A-storage C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

With expiration-based leases enabled, CockroachDB repeatedly overwrites the same set of keys (corresponding to a lease per KV range). With our existing skiplist memtable implementation, each duplicate key written will copy the entire key into the memtable arena. These new, frequent writes can quickly fill a memtable forcing a flush. Triggering early flushes hurts write bandwidth efficiency, because it makes it less likely that the raft log entries contained within the memtable will be truncated before the flush.

We could consider changes to our memtable implementation that reduce the memory overhead of duplicate keys.

There's a paper about [Trie Memtables in Cassandra](https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf) that may prove illustrative.

Jira issue: PEBBLE-218

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name a file, test, or entry point; start by locating the skiplist memtable implementation and understanding how duplicate keys occupy its arena. Read the linked Trie Memtables paper, then define and validate an approach that reduces duplicate-key memory overhead without worsening write or flush behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.