cockroachdb / cockroachdb/cockroach
drt/operations: implement heuristics to prevent OOD (out-of-disk) during long running operations
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
### Background
Recently, `drt-chaos` experienced an outage due to an online-restore job filling up disk on two different nodes. After troubleshooting, it became clear there are at least two different issues,
- an online-restore is not robust wrt OOD
- concurrent import/restore operations are not robust wrt OOD
The latter invariant is effectively broken by `tpcc_drop.sh` which periodically imports `tpcc` out-of-band wrt operations. That is, an operation such as (online) restore or import may be executing concurrently with `tpcc_drop.sh`. Even if a single import/restore is completely robust wrt OOD, concurrent ones aren't. The former might be improved/fixed by MMA [1].
There is potentially another issues, as discussed in [2], which may or may not be directly addressed by MMA. Since `drt-chaos` runs multiple stores (four), after a series of imports and restores, there was a significant disk utilization imbalance wherein only one store was getting (restored) data, while the other three had plenty of capacity.
### Plan
First, we need to ensure there is sufficient disk capacity before selecting an (online) restore or import operation. For safety, we could disallow concurrent import/restore and have additional safeguards around online restore. Second, we need to revisit the spec for this cluster [3]. Currently, it uses AZ localities with two nodes per AZ. If an entire AZ is low on disk capacity, decommissioning a node in that AZ can prove challenging/infeasible.
[1] https://cockroachlabs.atlassian.net/browse/CRDB-55052
[2] https://cockroachlabs.slack.com/archives/C03JCUUSCD6/p1768537472389449
[3] https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/drtprod/configs/drt_chaos.yaml
Jira issue: CRDB-59102
Contributor guide
Assessment
This issue has not been assessed yet.