cockroachdb / cockroachdb/cockroach

kvprober pages SRE in case of range-level backpressure / overload, which is a false positive

Open
#147,346 1 comment 0 reactions 0 assignees View on GitHub
A-kv-observability C-bug O-sre
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Large workload ran on cluster, leading to range-level backpressure.

```
applying backpressure to limit range growth on batch ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›],... 75 skipped ..., ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], ConditionalPut [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], [txn: bd5fc066], [can-forward-ts]
```

```
6 appending new entries to log would exceed uncommitted entry size limit; dropping proposal
```

Therefore breaker tripped:

```
breaker: tripped with error: replica unavailable: (n1,s1):6 unable to serve request to r86:‹×› [(n4,s4):11, (n5,s5):17, (n3,s3):10, (n9,s9):8, (n1,s1):6, next=18, gen=100]: closed timestamp: 1747854016.988070865,0 (2025-05-21 19:00:16); raft status: {"id":"6","term":33,"vote":"6","commit":417907,"lead":"6","leadEpoch":"27","raftState":"StateLeader","applied":417907,"progress":{"6":{"match":417907,"next":418548,"state":"StateReplicate"},"8":{"match":417907,"next":417908,"state":"StateReplicate"},"a":{"match":417907,"next":417908,"state":"StateReplicate"},"b":{"match":417907,"next":417908,"state":"StateReplicate"},"11":{"match":417907,"next":417908,"state":"StateReplicate"}},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal HeartbeatTxn [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], [txn: aaf0f26b]
```

Therefore kvprober errors:

```
kv.Txn(Put(/Local/Range/Table/111/1/‹×›/‹×›); Del(-)), r=86 failed with: replica unavailable: (n1,s1):6 unable to serve request to r86:‹×› [(n4,s4):11, (n5,s5):17, (n3,s3):10, (n9,s9):8, (n1,s1):6, next=18, gen=100]: closed timestamp: 1747853787.026100543,0 (2025-05-21 18:56:27); raft status: {"id":"6","term":33,"vote":"6","commit":359739,"lead":"6","leadEpoch":"27","raftState":"StateLeader","applied":359739,"progress":{"6":{"match":359739,"next":360252,"state":"StateReplicate"},"8":{"match":359739,"next":359740,"state":"StateReplicate"},"a":{"match":359739,"next":359740,"state":"StateReplicate"},"b":{"match":359739,"next":359740,"state":"StateReplicate"},"11":{"match":359739,"next":359740,"state":"StateReplicate"}},"leadtransferee":"0"}: have been waiting 79.50s for slow proposal EndTxn(abort) [/Table/111/1/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›/‹×›], [txn: 430f9dc4], [can-forward-ts] [quarantined=false]
```

All was on a single range. DB behavior looked good, as cluster was healthy overall, due to backpressure + AC protecting cluster.

OTOH, kvprober behavior not ideal. Ideal kvprober behavior is to page in case of bad DB behavior but not overload, since typical cause of overload is too much workload. SRE cannot action this problem. Customer needs to action instead, either by changing cluster size or reducing workload.

We implement don't page SRE in case of overload by having kvprober skip the the admission control queues. But that doesn't work for these range-level overload problem. This ticket tracks adjust kvprober to not fire in case of range-level overload also.

Range-level backpressure leading to kvprober pages doesn't happen very often in CC. Perhaps it is not worth closing this ticket, until it starts happening more often.

**To Reproduce**

I do not have a reproduction at this time.

**Expected behavior**

Range-level backpressure caused kvprober pages, even tho goal of kvprober is not page in case of overload due to too much workload. See above for more.

**Additional data / screenshots**
N/A

**Environment:**
All versions

**Additional context**
https://cockroachlabs.atlassian.net/browse/CLOUDOPS-14200

Jira issue: CRDB-51011

Contributor guide

Open the contributing guide

Research direction

No reproduction, file, entry point, or test is provided. Start by locating kvprober's admission-control handling and the code that turns range-level backpressure or replica-unavailable errors into SRE pages; done means overload caused by workload does not page SRE while genuine database failures still do, with regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.