cockroachdb / cockroachdb/cockroach

kv: unnecessary DistSender delay after range descriptor change

Open
#133,178 1 comment 0 reactions 0 assignees View on GitHub
branch-master C-bug O-perturbation T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

If a replica and lease transfer, so a client node has a stale cache, the node will wait an unnecessary 50ms before retrying with the updated range descriptor. This causes a latency spike during a decommision and other range transfers.

**To Reproduce**

This occured with the following run:

`COCKROACH_RANDOM_SEED=6457058786027231088 roachtest run perturbation/metamorphic/decommission`

The test configuration is:
```
2024/10/22 17:18:55 admission_control_latency.go:741: test variations are:
seed: 6457058786027231088,
fillDuration: 10m0s,
maxBlockBytes: 1,
perturbationDuration: 30m0s,
validationDuration: 5m0s,
ratioOfMax: 0.500000,
splits: 10000,
numNodes: 12,
numWorkloadNodes: 1,
vcpu: 8,
disks: 1,
memory: high,
leaseType: expiration,
cloud: gce,
perturbation: decommission{drain: false}
```

**Expected behavior**
DistSender should detect a failure due to a stale cache and retry immediately after updating the cache. One easy option is to drop the `DefaultRetryOptions` for DistSender to have a shorter initial delay before retrying. Setting this at 10ms would likely be enough to mask this failure.

**Additional data / screenshots**
Here is a trace from that run. Notice the following lines:

[2024-10-22T12_58_40Z-1014272887179870218.zip](https://github.com/user-attachments/files/17480610/2024-10-22T12_58_40Z-1014272887179870218.zip)

```
6.629ms 0.132ms event:kv/kvclient/kvcoord/dist_sender.go:2132 [n10,client=10.142.1.83:34774,hostssl,user=roachprod,txn=fd8fa429] will retry after 45.697104ms
52.580ms 45.950ms event:kv/kvclient/kvcoord/dist_sender.go:2624 [n10,client=10.142.1.83:34774,hostssl,user=roachprod,txn=fd8fa429] r2021: sending batch 1 Get to (n11,s11):5
```

**Environment:**
- CockroachDB version 24.3.0 (but likely occurs in all prior versions as well).

Jira issue: CRDB-43494

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.