cockroachdb / cockroachdb/cockroach
backupccl: parallelize reading descriptors in SHOW BACKUP
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In https://cockroachlabs.atlassian.net/browse/CC-25383?focusedCommentId=195208 we found via traces that while reading of manifests is parallelized, we spend a large time sequentially reading `descriptorslist.sst` files. In an MR cluster where the backup bucket is far away from the gateway this is particularly exacerbated. We should investigate ways to parallelize this loop https://github.com/cockroachdb/cockroach/blob/ce64d8b2039eef214d6340a9c025194ef88cf021/pkg/ccl/backupccl/show.go#L778-L779 such that reading descriptors from GCS/AWS is all done concurrently, and we spill to temp disk while sequentially processing the descriptors during query execution. Out of a 30s query, these sequential reads took almost 20s.
Jira issue: CRDB-33031
Contributor guide
Assessment
This issue has not been assessed yet.