cockroachdb / cockroachdb/cockroach

sql: stats flushing can block node drain

Open
#138,047 4 comments 0 reactions 0 assignees View on GitHub
A-admission-control branch-master C-bug O-perturbation T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

A drain of a node was blocked during `PersistedSQLStats.MaybeFlush`. This prevented the drain from completing and the drain timed out after 10m and failed.

**To Reproduce**

This doesn't happen every time, but it occurs enough that it should be reproducible. I think this is due to having the disk bandwidth limiter enabled. I believe the sql stats flushing happens at elastic AC priority so if there is already an AC queue the drain command can block indefinitely.

The command to re-run this test configuration is:
```
COCKROACH_RANDOM_SEED=5641047121427902826 roachtest run perturbation/metamorphic/decommission
```

Note that this configuration has `diskBandwidthLimit: 350MiB` set which might be a major contributor to this problem.

**Expected behavior**
Drain should finish quickly. If we can't persist the sql stats in a short amount of time, then we should shutdown without waiting for them.

**Additional data / screenshots**
I captured a side-eye snapshot mid-drain to see what was blocked here: https://app.side-eye.io/#/snapshots/493
Note that it never completed the SQL drain.

A graph of the run can be seen here: https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?orgId=1&from=1735325171471&to=1735326812198&var-cluster=baptistcockroachlabscom-1735325015-01-n6cpu32sm&var-DS_PROMETHEUS=v9Zz2K6nz&var-node=All&var-rate_interval=1m&var-token_type=All

The drain starts at 19:02:03.

**Environment:**
- CockroachDB version [master]

**Additional context**
Blocked draining can cause a problem for customers since they expect drain to complete quickly.

Jira issue: CRDB-45867

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.