cockroachdb / cockroachdb/pebble

db: shared compaction concurrency limit across multiple Pebble instances

Open
#3,813 4 comments 0 reactions 1 assignee Claimed by @sumeerbhola View on GitHub
A-storage no-issue-activity T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

In CockroachDB multi-store deployments, especially with large numbers of stores (8 or more), the CPU consumed by compactions can be significant. We do need a per store compaction concurrency limit, since disk bandwidth is a per-store resource, but we should additionally have a shared compaction concurrency limiter.

This shared limiter should fairly adjudicate on which compaction gets to run next based on a score of how important it is. For example,
- level score driven compactions could be compared based on the score, or additionally take into account the number of sub-levels if this is a compaction out of L0.
- a read-driven compaction in one store is strictly less important than a score driven compaction in another store.
- delete-only and move compactions are cheap, so don't need to respect this concurrency limit.

Jira issue: PEBBLE-230

Epic CRDB-41111

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.