cockroachdb / cockroachdb/cockroach
kvserver: dead nodes' liveness not gossiped reliably on liveness lease acquisition
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
In https://github.com/cockroachdb/cockroach/pull/98150, we removed all but the first liveness gossip on liveness lease acquisition. This was thought to be equivalent to gossipping "everything" on extensions as well. However, it turns out it was not but we don't understand why:
https://github.com/cockroachdb/cockroach/issues/99268#issuecomment-1483064544
**To Reproduce**
See https://github.com/cockroachdb/cockroach/issues/99268#issuecomment-1483064544 which has SHAs on which this reproduced in the `replicate/wide` roachtest.
**Expected behavior**
Just works? Nodes 7-9 in `replicate/wide` ought to be available via gossip reliably on n1-n6 when they come back up.
Jira issue: CRDB-26066
Contributor guide
Research direction
Start with the liveness lease acquisition and gossip behavior described in PR 98150 and issue 99268, including its linked reproduction comment. Run or inspect the replicate/wide roachtest on the referenced SHAs to understand how nodes 7-9 become unavailable via gossip. Done means their liveness is reliably gossiped to nodes 1-6 when they return.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100