cockroachdb / cockroachdb/cockroach
sql: stats flushing can block node drain
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
A drain of a node was blocked during `PersistedSQLStats.MaybeFlush`. This prevented the drain from completing and the drain timed out after 10m and failed.
**To Reproduce**
This doesn't happen every time, but it occurs enough that it should be reproducible. I think this is due to having the disk bandwidth limiter enabled. I believe the sql stats flushing happens at elastic AC priority so if there is already an AC queue the drain command can block indefinitely.
The command to re-run this test configuration is:
```
COCKROACH_RANDOM_SEED=5641047121427902826 roachtest run perturbation/metamorphic/decommission
```
Note that this configuration has `diskBandwidthLimit: 350MiB` set which might be a major contributor to this problem.
**Expected behavior**
Drain should finish quickly. If we can't persist the sql stats in a short amount of time, then we should shutdown without waiting for them.
**Additional data / screenshots**
I captured a side-eye snapshot mid-drain to see what was blocked here: https://app.side-eye.io/#/snapshots/493
Note that it never completed the SQL drain.
A graph of the run can be seen here: https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?orgId=1&from=1735325171471&to=1735326812198&var-cluster=baptistcockroachlabscom-1735325015-01-n6cpu32sm&var-DS_PROMETHEUS=v9Zz2K6nz&var-node=All&var-rate_interval=1m&var-token_type=All
The drain starts at 19:02:03.
**Environment:**
- CockroachDB version [master]
**Additional context**
Blocked draining can cause a problem for customers since they expect drain to complete quickly.
Jira issue: CRDB-45867
Contributor guide
Assessment
This issue has not been assessed yet.