cockroachdb / cockroachdb/cockroach
kv: force flushed tokens from splits can impact latency
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Related to #137933, if there is a backlog of disk bandwidth tokens, and then a range operation like a split occurs, a large number of tokens are released at once and this can temporarily overload the nodes in the cluster.
**To Reproduce**
1) Run a KV workload at about 50% utilization on a 5 node cluster and fill for about 15 minutes.
2) Drain and decommission one of the nodes
3) During the decommission, there will be "step" functions in the number of outstanding tokens. When these occur latency will be impacted.
**Expected behavior**
Splits should occur without a large impact on performance. While there is a need to force flush some tokens to allow the split to occur, the amount being flushed causes a large performance hiccup.
**Additional data / screenshots**
This can be seen here:
https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?from=1734972000000&to=1734972300000&var-cluster=baptistcockroachlabscom-1734968054-01-n6cpu32sm&orgId=1&var-DS_PROMETHEUS=v9Zz2K6nz&var-node=All&var-rate_interval=1m&var-token_type=All
Notice that there are a large number of force flused tokens at 16:43 from n3's perspective.
This causes high KV latency on the recipient node (n4) and a large impact on latency:
**Environment:**
- CockroachDB version master
**Additional context**
The impact with disk bandwidth tokens and `apply_to_all` is much larger than acceptable.
Jira issue: CRDB-45814
Contributor guide
Assessment
This issue has not been assessed yet.