cockroachdb / cockroachdb/cockroach

backupccl: parallelize reading descriptors in SHOW BACKUP

Open
#113,511 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In https://cockroachlabs.atlassian.net/browse/CC-25383?focusedCommentId=195208 we found via traces that while reading of manifests is parallelized, we spend a large time sequentially reading `descriptorslist.sst` files. In an MR cluster where the backup bucket is far away from the gateway this is particularly exacerbated. We should investigate ways to parallelize this loop https://github.com/cockroachdb/cockroach/blob/ce64d8b2039eef214d6340a9c025194ef88cf021/pkg/ccl/backupccl/show.go#L778-L779 such that reading descriptors from GCS/AWS is all done concurrently, and we spill to temp disk while sequentially processing the descriptors during query execution. Out of a 30s query, these sequential reads took almost 20s.

Jira issue: CRDB-33031

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.