cockroachdb / cockroachdb/cockroach

bulkmerge: parallelize the final merge iteration

Open
#162,539 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

During the distributed merge pipeline, the final merge iteration (which writes temporary SSTs into KV) runs across multiple nodes, but on each node the work is currently executed in a single goroutine. This issue proposes parallelizing that work within a node.

Distributed merge already runs under admission control, so this should be safe from a resource perspective. The main expected benefit is for larger clusters that have enough CPU to take advantage of additional intra-node parallelism.

Jeff Swenson previously prototyped this approach, which we can use as a starting point for the change:
https://github.com/cockroachdb/cockroach/pull/162018/changes/cbb6728b133fd8c6a1bffaec53f45df02661f6f1

Jira issue: CRDB-59522

Epic CRDB-62564

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.