cockroachdb / cockroachdb/cockroach

kv: inefficient usage of Locality

Open
#114,612 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-bug O-23.2-scale-testing O-testcluster T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Looking at the CPU profile of a node in a large cluster, I noticed a significant fraction of time (>25%) spent in GC. Digging into the new allocations, it appears that a lot of the time is spent on Locality copying or calculations. This could be optimized in a number of ways.

image

[CPU profile](https://github.com/cockroachdb/cockroach/files/13383129/profile.4.pb.gz)

Digging into what was creating this amount of garbage using the "Heap (recent allocs)" tag showed a few unexpected places where we do allocations:
Specifically >25% of allocations (~37GB every 5s) are done so we can add the `node` tag to the Locality - https://github.com/cockroachdb/cockroach/commit/ed34965b3622d551cd56403838ebe3024036d848. There are a number of ways this could be done more efficiently.
image
[Heap (recent allocs)](https://github.com/cockroachdb/cockroach/files/13383145/profile.5.pb.gz)

Many of the allocations could likely be removed with only a small amount of work each.

If these are removed the impact could be larger than a 25% improvement in throughput since both the allocation and the GC will be remove. It could have an even bigger improvement in P99 latency since the GCs will become much less frequent and less impactful when they run.

**Expected behavior**
Much of the "hot path" for data is already optimized to minimize GC work. The few places that are showing up in the memory profiles are easier to fix and should be done.

**Additional data / screenshots**
This was on a test cluster running 23.2.alpha6.
There are 45 nodes, each with 16vCPU and 10 stores/node.
There are 2.2M ranges on this system so ~50K leases/store.

Jira issue: CRDB-33596

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.