cockroachdb / cockroachdb/cockroach

storage: smooth pebble compaction

Open
#133,948 4 comments 0 reactions 0 assignees View on GitHub
A-storage branch-master C-bug C-performance o-perf-efficiency T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In various scenarios, while pebble compactions (including memtable -> L0) are running the latency for operations increases significantly. This is particularly noticeable on small clusters, and most noticeable on a single node cluster. In these situations each node will oscillate between "fast" (not running compaction) and "slow" (running compaction) with periods on the order of 10-60s.

Proposed solution:
Attempt to smooth the compaction by detecting scenarios where there are idle periods between compactions. By smoothing the compaction, the system will always run at "medium" rate rather than oscillating between fast and slow. This has two primary benefits:
1) From the end user perspective, the latency is flatter.
2) The total throughput should increase for "open loop" workloads with fixed concurrency.

References:
[Internal slack discussion](https://cockroachlabs.slack.com/archives/C4X2J0RH6/p1730295002868889)
[Presentation on latency](https://drive.google.com/file/d/1OX1vQiPQYgiPGKPmIke244pp3ZYIB6U8/view?usp=sharing
)
https://github.com/cockroachdb/pebble/pull/2004

Jira issue: CRDB-43849

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.