cockroachdb / cockroachdb/cockroach

storage: `MVCCValueMerger` is allocation heavy

Open
#106,567 2 comments 0 reactions 0 assignees View on GitHub
A-storage C-performance T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In a high-throughput, write-heavy workload (`kv33/2kb`), we see that **22.5%** of go heap allocated memory (by size) come from `storage.MVCCValueMerger`.

This is expensive and puts pressure on the Go GC, leading to more frequent GCs and higher foreground tail latency. Can we improve it? Here are some ideas:
- [x] **(easy, high impact)** properly size `meta.RawBytes` in `MVCCValueMerger.Finish` and then `MarshalToSizedBuffer` into it instead of `append`
- [x] **(easy)** hang `merged roachpb.InternalTimeSeriesData` off `MVCCValueMerger` to escape to heap
- [x] **(easy)** hang `meta` enginepb.MVCCMetadataSubsetForMergeSerialization off `MVCCValueMerger` to escape to heap
- [x] **(easy)** don't let `timeSeriesOp` loop iteration variable escape, instead use `&t.timeSeriesOps[i]`
- [ ] **(medium)** re-use `MVCCValueMerger.meta.RawBytes` across calls to `deserializeMVCCValueAndAppend` by not resetting it in `protoutil.Unmarshal`
- [ ] **(hard)** batch allocate inner primitive slices in `InternalTimeSeriesData`
- [ ] **(medium)** pool MVCCValueMergers, returning to the pool in `Finish`

Screenshot 2023-07-11 at 1 04 58 AM

[heap_profile.pb.gz](https://github.com/cockroachdb/cockroach/files/12010865/heap_profile.pb.gz) (use the `alloc_space` sample)

Jira issue: CRDB-29617

Contributor guide

Open the contributing guide

Research direction

Start with the alloc_space heap profile and inspect storage.MVCCValueMerger, especially Finish and deserializeMVCCValueAndAppend, along with InternalTimeSeriesData. Choose one of the unchecked reuse, batching, or pooling ideas, then validate that the kv33/2kb workload shows lower allocation pressure without changing merge behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.