cockroachdb / cockroachdb/cockroach
kv: multistore uneven distribution of data during restore
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Full cluster restore on `drt-scale-restore` cluster is erroring out because of one of the store running out of disk capacity.
**Cluster specification:**
Number of nodes: 68
Number of store per node: 8, each of size 375GB.
Restore db size: 125-130TB.
During restore it is observed that data is being distributed to very few nodes out of 68. The nodes on which data is restored used only one store out of 8.
Datadog disk [usage dashboard](https://us5.datadoghq.com/dashboard/pbe-ic2-3qt/drt?fromUser=true&fullscreen_end_ts=1731416037778&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1731395480604&fullscreen_widget=8090532782126254&refresh_mode=sliding&tpl_var_cluster%5B0%5D=drt-scale-restore&tpl_var_workload_cluster%5B0%5D=workload-scale-restore&from_ts=1731402920000&to_ts=1731413720000&live=true).
Jira issue: CRDB-44315
Epic CRDB-41111
Contributor guide
Assessment
This issue has not been assessed yet.