cockroachdb / cockroachdb/cockroach

kv: allocator makes sub-optimal decisions at beginning of benchmark run (3% throughput TPC-C)

Open
#141,867 2 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement o-perf-efficiency
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

On a 5 node TPC-C cluster which is doing repeated 10 minute runs, we find that between 10% and 20% of the runs have sub-optimal performance (when compared with the other runs). The problem seems to be uneven CPU usage which can be traced back to allocator placement decisions. A good run's CPU usage will look something like this, where the CPU is _relatively_ evenly distributed between nodes:

![Image](https://github.com/user-attachments/assets/cb911e65-56e1-4162-b231-1cb5bb5cd028)

A bad run looks like this (where one node has significantly lower CPU utilization):

![Image](https://github.com/user-attachments/assets/a46ff0f3-6e81-4a09-a7c0-ad9e7c6f21e4)

Through some experimentation, we've found that if we tune the cluster settings as follows:

```
set cluster setting kv.allocator.load_based_rebalancing_interval = '10s';
set cluster setting kv.allocator.store_cpu_rebalance_threshold = 0.01;
```

Performance across all runs gets better, the CPU utilization is much more uniform throughout the runs, and throughput improves by 3%.

![Image](https://github.com/user-attachments/assets/9c43cd21-cfe3-4ea6-bb39-928251f248f3)

While these settings may not be appropriate for longer runs (or permanently in production), the allocator could benefit from being far more aggressive in the first N minutes after a cluster is first started (or a dramatic workload shift occurs).

Jira issue: CRDB-48038

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.