cockroachdb / cockroachdb/cockroach
bulkmerge: parallelize the final merge iteration
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
During the distributed merge pipeline, the final merge iteration (which writes temporary SSTs into KV) runs across multiple nodes, but on each node the work is currently executed in a single goroutine. This issue proposes parallelizing that work within a node.
Distributed merge already runs under admission control, so this should be safe from a resource perspective. The main expected benefit is for larger clusters that have enough CPU to take advantage of additional intra-node parallelism.
Jeff Swenson previously prototyped this approach, which we can use as a starting point for the change:
https://github.com/cockroachdb/cockroach/pull/162018/changes/cbb6728b133fd8c6a1bffaec53f45df02661f6f1
Jira issue: CRDB-59522
Epic CRDB-62564
Contributor guide
Assessment
This issue has not been assessed yet.