cockroachdb / cockroachdb/cockroach
kv: unnecessary DistSender delay after range descriptor change
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
If a replica and lease transfer, so a client node has a stale cache, the node will wait an unnecessary 50ms before retrying with the updated range descriptor. This causes a latency spike during a decommision and other range transfers.
**To Reproduce**
This occured with the following run:
`COCKROACH_RANDOM_SEED=6457058786027231088 roachtest run perturbation/metamorphic/decommission`
The test configuration is:
```
2024/10/22 17:18:55 admission_control_latency.go:741: test variations are:
seed: 6457058786027231088,
fillDuration: 10m0s,
maxBlockBytes: 1,
perturbationDuration: 30m0s,
validationDuration: 5m0s,
ratioOfMax: 0.500000,
splits: 10000,
numNodes: 12,
numWorkloadNodes: 1,
vcpu: 8,
disks: 1,
memory: high,
leaseType: expiration,
cloud: gce,
perturbation: decommission{drain: false}
```
**Expected behavior**
DistSender should detect a failure due to a stale cache and retry immediately after updating the cache. One easy option is to drop the `DefaultRetryOptions` for DistSender to have a shorter initial delay before retrying. Setting this at 10ms would likely be enough to mask this failure.
**Additional data / screenshots**
Here is a trace from that run. Notice the following lines:
[2024-10-22T12_58_40Z-1014272887179870218.zip](https://github.com/user-attachments/files/17480610/2024-10-22T12_58_40Z-1014272887179870218.zip)
```
6.629ms 0.132ms event:kv/kvclient/kvcoord/dist_sender.go:2132 [n10,client=10.142.1.83:34774,hostssl,user=roachprod,txn=fd8fa429] will retry after 45.697104ms
52.580ms 45.950ms event:kv/kvclient/kvcoord/dist_sender.go:2624 [n10,client=10.142.1.83:34774,hostssl,user=roachprod,txn=fd8fa429] r2021: sending batch 1 Get to (n11,s11):5
```
**Environment:**
- CockroachDB version 24.3.0 (but likely occurs in all prior versions as well).
Jira issue: CRDB-43494
Contributor guide
Assessment
This issue has not been assessed yet.