cockroachdb / cockroachdb/cockroach

storage: allow configuration of memtable size

Open
#118,661 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-enhancement T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Today a Pebble memtable is always 64MB. The size of the memtable impacts write bandwidth, because the longer a flush is delayed the more likely raft log truncation is able to elide keys before they're written to disk. A similar dynamic applies to expiration-based leases which repeatedly overwrite the same set of keys.

Some customers' environments could easily tolerate significantly larger memtables. Allocation of a new memtable reserves space from the block cache, so as long as the block cache is sufficiently large there is not significant risk exhausting memory.

I'm not sure how we would want to expose this. Some clusters run on heterogeneous hardware, and a particular memtable size might not be appropriate for all stores. Additionally some clusters run many stores per node, so an increase in memtable size is multiplied across the configured stores. Some ability to configure memtable size may make it more feasible for large customers to make use of expiration-based leases.

Jira issue: CRDB-35825

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.