cockroachdb / cockroachdb/cockroach

kv: multistore uneven distribution of data during restore

Open
#134,960 8 comments 0 reactions 0 assignees View on GitHub
branch-master branch-release-24.3 C-bug C-enhancement O-24.3-scale-testing O-support O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Full cluster restore on `drt-scale-restore` cluster is erroring out because of one of the store running out of disk capacity.

**Cluster specification:**
Number of nodes: 68
Number of store per node: 8, each of size 375GB.
Restore db size: 125-130TB.

During restore it is observed that data is being distributed to very few nodes out of 68. The nodes on which data is restored used only one store out of 8.

Datadog disk [usage dashboard](https://us5.datadoghq.com/dashboard/pbe-ic2-3qt/drt?fromUser=true&fullscreen_end_ts=1731416037778&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1731395480604&fullscreen_widget=8090532782126254&refresh_mode=sliding&tpl_var_cluster%5B0%5D=drt-scale-restore&tpl_var_workload_cluster%5B0%5D=workload-scale-restore&from_ts=1731402920000&to_ts=1731413720000&live=true).

Screenshot 2024-11-12 at 3 24 42 PM

Jira issue: CRDB-44315

Epic CRDB-41111

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.