cockroachdb / cockroachdb/cockroach

kvclient: default distsender backoff is large for modern clusters

Open
#173,370 3 comments 0 reactions 0 assignees View on GitHub
A-kv-client branch-master C-bug O-support T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The default retry options are:

```golang
// DefaultRetryOptions should be used for retrying most
// network-dependent operations.
func DefaultRetryOptions() retry.Options {
// TODO(bdarnell): This should vary with network latency.
// Derive the retry options from a configured or measured
// estimate of latency.
return retry.Options{
InitialBackoff: 50 * time.Millisecond,
MaxBackoff: 1 * time.Second,
Multiplier: 2,
}
}
```

The initial backoff may be an > 10x the RTT in many clusters. This makes issues like #173342 very impactful. We should revisit this TODO.

Jira issue: CRDB-66685

Contributor guide

Open the contributing guide

Research direction

Start at the DefaultRetryOptions function shown in the issue and trace how distsender uses the returned retry.Options. Investigate the related behavior in issue #173342 and determine how configured or measured network latency should affect the initial backoff. Done means the retry behavior is appropriate for modern cluster RTTs, with tests covering the revised defaults or latency-dependent behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.