cockroachdb / cockroachdb/cockroach
kv: allocator makes sub-optimal decisions at beginning of benchmark run (3% throughput TPC-C)
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
On a 5 node TPC-C cluster which is doing repeated 10 minute runs, we find that between 10% and 20% of the runs have sub-optimal performance (when compared with the other runs). The problem seems to be uneven CPU usage which can be traced back to allocator placement decisions. A good run's CPU usage will look something like this, where the CPU is _relatively_ evenly distributed between nodes:

A bad run looks like this (where one node has significantly lower CPU utilization):

Through some experimentation, we've found that if we tune the cluster settings as follows:
```
set cluster setting kv.allocator.load_based_rebalancing_interval = '10s';
set cluster setting kv.allocator.store_cpu_rebalance_threshold = 0.01;
```
Performance across all runs gets better, the CPU utilization is much more uniform throughout the runs, and throughput improves by 3%.

While these settings may not be appropriate for longer runs (or permanently in production), the allocator could benefit from being far more aggressive in the first N minutes after a cluster is first started (or a dramatic workload shift occurs).
Jira issue: CRDB-48038
Contributor guide
Assessment
This issue has not been assessed yet.