cockroachdb / cockroachdb/cockroach

kvserver: dead nodes' liveness not gossiped reliably on liveness lease acquisition

Open
#99,652 4 comments 0 reactions 0 assignees View on GitHub
C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In https://github.com/cockroachdb/cockroach/pull/98150, we removed all but the first liveness gossip on liveness lease acquisition. This was thought to be equivalent to gossipping "everything" on extensions as well. However, it turns out it was not but we don't understand why:

https://github.com/cockroachdb/cockroach/issues/99268#issuecomment-1483064544

**To Reproduce**

See https://github.com/cockroachdb/cockroach/issues/99268#issuecomment-1483064544 which has SHAs on which this reproduced in the `replicate/wide` roachtest.

**Expected behavior**

Just works? Nodes 7-9 in `replicate/wide` ought to be available via gossip reliably on n1-n6 when they come back up.

Jira issue: CRDB-26066

Contributor guide

Open the contributing guide

Research direction

Start with the liveness lease acquisition and gossip behavior described in PR 98150 and issue 99268, including its linked reproduction comment. Run or inspect the replicate/wide roachtest on the referenced SHAs to understand how nodes 7-9 become unavailable via gossip. Done means their liveness is reliably gossiped to nodes 1-6 when they return.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.