cockroachdb / cockroachdb/cockroach
rangefeed: mux rangefeeds should move away from stuck replicas
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
For legacy rangefeeds have a `stuckRangeFeedCanceler` that attempts to detect and move replicas that haven't received updates for some time. A similar approach should be implemented for mux rangefeeds as well. Rangefeeds can be stuck if a node is partitioned from the leader and therefore can't get closed timestamp updates.
**To Reproduce**
The test `TestPartialPartition` will reproduce this behavior if `rangeFeedsEnabled` is set to true. Specifically a rangefeed that is on a node partitioned from the leader will start logging `RangeFeed closed timestamp xxx is behind by yyy` but the client side will not do anything.
**Expected behavior**
Rangefeeds can't get stuck because the are critical for internal tasks to work correctly. We need a mechanism to unstuck these situations.
**Potential solutions**
This can be handled on either the client or the server side. On the client side we could introduce a `stuckRangeFeedCanceler` for mux rangefeeds that will move to the next replica. On the server side we could cancel rangefeeds inside `handleClosedTimestampUpdateRaftMuLocked` to close the gRPC stream if a timeout is hit. We might want to do both (short server timeout and longer client timeout).
Jira issue: CRDB-36251
Contributor guide
Assessment
This issue has not been assessed yet.