cockroachdb / cockroachdb/cockroach

rangefeed: mux rangefeeds should move away from stuck replicas

Open
#119,490 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed C-bug T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

For legacy rangefeeds have a `stuckRangeFeedCanceler` that attempts to detect and move replicas that haven't received updates for some time. A similar approach should be implemented for mux rangefeeds as well. Rangefeeds can be stuck if a node is partitioned from the leader and therefore can't get closed timestamp updates.

**To Reproduce**

The test `TestPartialPartition` will reproduce this behavior if `rangeFeedsEnabled` is set to true. Specifically a rangefeed that is on a node partitioned from the leader will start logging `RangeFeed closed timestamp xxx is behind by yyy` but the client side will not do anything.

**Expected behavior**
Rangefeeds can't get stuck because the are critical for internal tasks to work correctly. We need a mechanism to unstuck these situations.

**Potential solutions**
This can be handled on either the client or the server side. On the client side we could introduce a `stuckRangeFeedCanceler` for mux rangefeeds that will move to the next replica. On the server side we could cancel rangefeeds inside `handleClosedTimestampUpdateRaftMuLocked` to close the gRPC stream if a timeout is hit. We might want to do both (short server timeout and longer client timeout).

Jira issue: CRDB-36251

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.