cockroachdb / cockroachdb/cockroach
kvserver: store out of disk while being streamed raft recovery snapshots due to earlier splits
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Node 75 has gone down in drt-scale cluster
The node went down due to disk full. The behaviour is strange as only one store capacity went to 100%.
Here is the changes that caused the change in the sizes:
* At 5:20 AM, we dropped DB of size 123TB, thats why the capacity came down at 6:30AM.
* At 12:30, we started the import for a DB of size 68TB which finished at 16:25.
* At 16:38, we dropped another DB of 7TB capacity. CG completed by 21:55.
* At 16:58, we started import of a DB of 570TB capacity which was complete by 18:28.
* Post this, TPCC workload was running and changefeeds were started.
The graph shows that around 12:00 PM, the capacity of store 298 went down and then kept increasing till 100%. The overall capacity usage of the cluster is only around 60%.
Debug zip location - https://console.cloud.google.com/storage/browser/_details/150-node-cluster-debug/debug-zips/2024_11_07_02_35_20.zip;tab=live_object?project=cockroach-drt.
Slack thread - https://cockroachlabs.slack.com/archives/C07HPMBLVJ7/p1730932142869589
Datadog link - [https://us5.datadoghq.com/dashboard/pbe-ic2-3qt/drt?fromUser=true&refresh_mode=paused[…]d-scale&from_ts=1730872800000&to_ts=1730933220000&live=false](https://us5.datadoghq.com/dashboard/pbe-ic2-3qt/drt?fromUser=true&refresh_mode=paused&tpl_var_host%5B0%5D=drt-scale-0075&tpl_var_workload_cluster%5B0%5D=workload-scale&from_ts=1730872800000&to_ts=1730933220000&live=false)
![Uploading storage_graph.png…]()
Jira issue: CRDB-44105
Contributor guide
Assessment
This issue has not been assessed yet.