cockroachdb / cockroachdb/cockroach

storage: attribute compaction load to a replica

Open
#98,377 3 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution A-storage C-enhancement T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

CPU replica and lease rebalancing was added as an option in https://github.com/cockroachdb/cockroach/issues/90582. The attribution of CPU to a replica is important, in order to correctly estimate the impact of a rebalancing action. The accuracy of this estimate determines the quality of placement decisions.

Currently, CPU is attributed to Replicas for Raft and Requests https://github.com/cockroachdb/cockroach/pull/92858. It is desirable to also attribute compaction CPU, compaction bytes read and compaction bytes written involving the keyspace of the range a replica belongs to.

The graph below shows the CPU profile of a node servicing a heavy write workload. Compaction represents 28% of sampled CPU time.

![image](https://user-images.githubusercontent.com/39606633/224366801-91efc7c6-aef2-4c6e-b5bc-4ae4d52f5809.png)

**Describe the solution you'd like**

Attribute the following metrics to a replica:
- CPU time spend on a compaction
- Bytes read during compaction
- Bytes written during compaction

**Additional context**
Just CPU is sufficient to close this issue, the other two are desirable but not a strict requirement.

From @petermattis

> Pebble knows which keys it is compacting, so this seems feasible. Heck, you could approximate this now using the Pebble Event callbacks that tell you when a compaction is finished and using the bounds of the compaction to estimate how much work was done by ranges that overlap those compaction bounds.

Jira issue: CRDB-25236

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing replica attribution for Raft and Requests described in PR 92858. Then inspect the Pebble compaction event callbacks and how compaction key bounds relate to ranges; done means CPU time for compaction is attributed to the affected replicas, with bytes read and written as optional follow-up metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.