cockroachdb / cockroachdb/pebble
db: shared compaction concurrency limit across multiple Pebble instances
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
In CockroachDB multi-store deployments, especially with large numbers of stores (8 or more), the CPU consumed by compactions can be significant. We do need a per store compaction concurrency limit, since disk bandwidth is a per-store resource, but we should additionally have a shared compaction concurrency limiter.
This shared limiter should fairly adjudicate on which compaction gets to run next based on a score of how important it is. For example,
- level score driven compactions could be compared based on the score, or additionally take into account the number of sub-levels if this is a compaction out of L0.
- a read-driven compaction in one store is strictly less important than a score driven compaction in another store.
- delete-only and move compactions are cheap, so don't need to respect this concurrency limit.
Jira issue: PEBBLE-230
Epic CRDB-41111
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.