cockroachdb / cockroachdb/pebble
db: consider memtable optimizations for duplicate keys
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
With expiration-based leases enabled, CockroachDB repeatedly overwrites the same set of keys (corresponding to a lease per KV range). With our existing skiplist memtable implementation, each duplicate key written will copy the entire key into the memtable arena. These new, frequent writes can quickly fill a memtable forcing a flush. Triggering early flushes hurts write bandwidth efficiency, because it makes it less likely that the raft log entries contained within the memtable will be truncated before the flush.
We could consider changes to our memtable implementation that reduce the memory overhead of duplicate keys.
There's a paper about [Trie Memtables in Cassandra](https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf) that may prove illustrative.
Jira issue: PEBBLE-218
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue does not name a file, test, or entry point; start by locating the skiplist memtable implementation and understanding how duplicate keys occupy its arena. Read the linked Trie Memtables paper, then define and validate an approach that reduces duplicate-key memory overhead without worsening write or flush behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100