cockroachdb / cockroachdb/cockroach
kvserver: during a partial partition, non-leaders wait 4s for a lease acquisition timeout
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
This applies once #117340 is merged.
During a partial partition, until replica circuit breakers kick in, nodes that are partitioned away from the leader/leaseholder will attempt to acquire the lease and wait 4 seconds before responding with an error.
I'm not sure what the solution will look like, but preferably they can fail fast without waiting for a timeout. I'm not sure if this is something we would require to do at the Raft layer or if we can simply detect that we are not in a state that it makes sense to campaign from.
**To Reproduce**
Create a 3-node cluster. Start a workload that sends SQL to all nodes. Create a partial partition between two of the nodes. Notice the 4-second latency on requests that persists for ~2 minutes.
**Expected behavior**
In this case the server (n3) knows it doesn't have a connection to n2 and it should either not attempt to acquire the lease or fail immediately if possible. An alternative is for the dist_sender to not send the message to itself, but as it doesn't know if there is a partial partition or n2 is completely down this is more difficult. After the first 2 minutes, the 4 second spikes mostly go away, but they still occur intermittently.
**Additional data / screenshots**
This is what the SQL latency looks like immediately after the partition is created between n2 and n3.
Here is a snippet of a trace from a slow request (request is from n3). The connection between n2 and n3 is broken.
```
0.537ms 0.015ms === operation:dist sender send _verbose:1 node:3 client:10.142.0.99:59180 hostnossl: user:root txn:1d077ad0
...
0.675ms 0.036ms event:kv/kvclient/kvcoord/dist_sender.go:2415 [n3,client=10.142.0.99:59180,hostnossl,user=root,txn=1d077ad0] attempting request with sorted replicas [(n2,s2):2 (n3,s3):3 (n1,s1):1] and lease (n2,s2):2
0.703ms 0.029ms event:kv/kvclient/kvcoord/dist_sender.go:2464 [n3,client=10.142.0.99:59180,hostnossl,user=root,txn=1d077ad0] r113: sending batch 1 Put, 1 EndTxn to (n3,s3):3
0.753ms 0.050ms event:kv/kvclient/kvcoord/dist_sender.go:2546 [n3,client=10.142.0.99:59180,hostnossl,user=root,txn=1d077ad0] skip proxy - don't proxy through self Put [/Table/106/1/6804694823852962294/0], EndTxn(parallel commit) [/Table/106/1/6804694823852962294/0], [txn: 1d077ad0], [can-forward-ts]
0.773ms 0.020ms event:rpc/nodedialer/nodedialer.go:138 [n3,client=10.142.0.99:59180,hostnossl,user=root,txn=1d077ad0] sending request to local client
0.788ms 0.015ms event:kv/kvclient/kvcoord/transport.go:213 [n3,client=10.142.0.99:59180,hostnossl,user=root,txn=1d077ad0] sending batch request
0.794ms 0.006ms === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:3 span.kind:server request:Put [/Table/106/1/6804694823852962294/0], EndTxn(parallel commit) [/Table/106/1/6804694823852962294/0], [txn: 1d077ad0], [can-forward-ts]
0.794ms 0.000ms [executeWriteBatch: {count: 1, duration 16µs}]
0.803ms 0.009ms event:server/node.go:1375 [n3] node received request: 1 Put, 1 EndTxn
0.830ms 0.027ms event:kv/kvserver/store_send.go:154 [n3,s3] executing Put [/Table/106/1/6804694823852962294/0], EndTxn(parallel commit) [/Table/106/1/6804694823852962294/0], [txn: 1d077ad0], [can-forward-ts]
0.842ms 0.012ms event:kv/kvserver/replica_send.go:184 [n3,s3,r113/3:/Table/106/1/6{66639…-84903…}] read-write path
...
0.926ms 0.047ms event:kv/kvserver/replica_send.go:521 [n3,s3,r113/3:/Table/106/1/6{66639…-84903…}] concurrency retry error: invalid lease
0.962ms 0.036ms event:kv/kvserver/replica_range_lease.go:1371 [n3,s3,r113/3:/Table/106/1/6{66639…-84903…}] request range lease (attempt #1)
4001.268ms 4000.306ms event:kv/kvserver/replica_range_lease.go:1479 [n3,s3,r113/3:/Table/106/1/6{66639…-84903…}] lease acquisition failed: context deadline exceeded
4001.337ms 0.069ms event:kv/kvserver/replica_send.go:199 [n3,s3,r113/3:/Table/106/1/6{66639…-84903…}] replica.Send got error: [NotLeaseHolderError] lease acquisition canceled because context canceled; r113: replica (n3,s3):3 not lease holder; lease holder unknown
```
Jira issue: CRDB-36460
Contributor guide
Assessment
This issue has not been assessed yet.