cockroachdb / cockroachdb/cockroach
storage: `MVCCValueMerger` is allocation heavy
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In a high-throughput, write-heavy workload (`kv33/2kb`), we see that **22.5%** of go heap allocated memory (by size) come from `storage.MVCCValueMerger`.
This is expensive and puts pressure on the Go GC, leading to more frequent GCs and higher foreground tail latency. Can we improve it? Here are some ideas:
- [x] **(easy, high impact)** properly size `meta.RawBytes` in `MVCCValueMerger.Finish` and then `MarshalToSizedBuffer` into it instead of `append`
- [x] **(easy)** hang `merged roachpb.InternalTimeSeriesData` off `MVCCValueMerger` to escape to heap
- [x] **(easy)** hang `meta` enginepb.MVCCMetadataSubsetForMergeSerialization off `MVCCValueMerger` to escape to heap
- [x] **(easy)** don't let `timeSeriesOp` loop iteration variable escape, instead use `&t.timeSeriesOps[i]`
- [ ] **(medium)** re-use `MVCCValueMerger.meta.RawBytes` across calls to `deserializeMVCCValueAndAppend` by not resetting it in `protoutil.Unmarshal`
- [ ] **(hard)** batch allocate inner primitive slices in `InternalTimeSeriesData`
- [ ] **(medium)** pool MVCCValueMergers, returning to the pool in `Finish`
[heap_profile.pb.gz](https://github.com/cockroachdb/cockroach/files/12010865/heap_profile.pb.gz) (use the `alloc_space` sample)
Jira issue: CRDB-29617
Contributor guide
Research direction
Start with the alloc_space heap profile and inspect storage.MVCCValueMerger, especially Finish and deserializeMVCCValueAndAppend, along with InternalTimeSeriesData. Choose one of the unchecked reuse, batching, or pooling ideas, then validate that the kv33/2kb workload shows lower allocation pressure without changing merge behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100