cockroachdb / cockroachdb/cockroach

kvserver: investigate and document non-cooperative lease acquisition timeline

Open
#103,656 0 comments 0 reactions 0 assignees View on GitHub
O-qa T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The timeline for non-cooperative lease acquisitions, and thus the unavailability distribution for various failure modes, can be hard to reason about due to the many factors involved. We should investigate and document this, and compare it to empirical unavailability observations (see #103654). This involves e.g.:

* Lease duration
* Lease renewal intervals
* Raft election timeouts (randomized)
* Raft heartbeat intervals
* Raft reproposal timeouts (for lease proposals)
* RPC heartbeat timeouts
* RPC circuit breaker policy
* gRPC keepalive and network timeouts
* TCP network and connection timeouts
* DistSender cache invalidation
* DistSender replica ordering and retry policy
* And probably more

There's a basic, incomplete timeline here:

https://github.com/cockroachdb/cockroach/blob/735cf2103246194a98899008ec9d80b110b083e5/pkg/base/config.go#L171-L182

Jira issue: CRDB-28114

Contributor guide

Open the contributing guide

Research direction

Start with the incomplete timeline in pkg/base/config.go around lines 171-182, then trace the listed lease, Raft, RPC, gRPC, TCP, and DistSender factors. Compare the resulting non-cooperative lease-acquisition timeline with the empirical unavailability observations in issue #103654; done means documenting the timeline and failure-mode distribution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grpc
Domain
databases, distributed-systems
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.