cockroachdb / cockroachdb/cockroach

kv: decommission takes a long time to complete with disk bandwidth tokens

Open
#137,933 1 comment 0 reactions 0 assignees View on GitHub
A-replication-admission-control-v2 A-storage branch-master C-bug O-perturbation T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

A cluster running at a low utilization can take hours to complete a small decommission if `kvadmission.store.provisioned_bandwidth` is set. Even if it is set above the rate that the cluster ever hits.

**To Reproduce**

1) Run a KV workload at about 50% utilization on a 5 node cluster and fill for about 15 minutes.
2) Drain and decommission one of the nodes
3) Note that the decommission takes over 2 hours to complete.

This can be reproduced with the following perturbation test command:

`COCKROACH_RANDOM_SEED=5641047121427902826 roachtest run perturbation/metamorphic/decommission`

**Expected behavior**
The decommission is expected to take slightly longer with the disk bandwidth tokens, however it should not take more than 2x as long. Also note that without the disk bandwidth tokens the impact of the decommission is smaller in general. To run without disk bandwidth tokens run the following:

`COCKROACH_RANDOM_SEED=5641047121427902826 PERTURBATION_OVERRIDE=diskBandwidthLimit=0 roachtest run perturbation/metamorphic/decommission`

**Additional data / screenshots**
The send queue size is growing during the entire test run, even before the decommission starts. There is a separate issue related to the "down steps" that are seen during the decommission as each of them causes a noticeable performance impact. Notice that the disk bandwidth used during the test stays below the 350MiB configured limit for almost the entire run, however it is still under-admitting.

https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?from=1734938499384&to=1734945532802&var-cluster=teamcity-18273707-1734936887-03-n6cpu32sm&orgId=1

image
image

**Environment:**
- CockroachDB version master

**Additional context**
What was the impact?
Since `kvadmission.store.provisioned_bandwidth` is not set by default this shouldn't block any releases, but it should block the setting of this setting for customers.

Jira issue: CRDB-45813

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.