cockroachdb / cockroachdb/cockroach
kv: inefficient usage of Locality
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Looking at the CPU profile of a node in a large cluster, I noticed a significant fraction of time (>25%) spent in GC. Digging into the new allocations, it appears that a lot of the time is spent on Locality copying or calculations. This could be optimized in a number of ways.
[CPU profile](https://github.com/cockroachdb/cockroach/files/13383129/profile.4.pb.gz)
Digging into what was creating this amount of garbage using the "Heap (recent allocs)" tag showed a few unexpected places where we do allocations:
Specifically >25% of allocations (~37GB every 5s) are done so we can add the `node` tag to the Locality - https://github.com/cockroachdb/cockroach/commit/ed34965b3622d551cd56403838ebe3024036d848. There are a number of ways this could be done more efficiently.
[Heap (recent allocs)](https://github.com/cockroachdb/cockroach/files/13383145/profile.5.pb.gz)
Many of the allocations could likely be removed with only a small amount of work each.
If these are removed the impact could be larger than a 25% improvement in throughput since both the allocation and the GC will be remove. It could have an even bigger improvement in P99 latency since the GCs will become much less frequent and less impactful when they run.
**Expected behavior**
Much of the "hot path" for data is already optimized to minimize GC work. The few places that are showing up in the memory profiles are easier to fix and should be done.
**Additional data / screenshots**
This was on a test cluster running 23.2.alpha6.
There are 45 nodes, each with 16vCPU and 10 stores/node.
There are 2.2M ranges on this system so ~50K leases/store.
Jira issue: CRDB-33596
Contributor guide
Assessment
This issue has not been assessed yet.