cockroachdb / cockroachdb/cockroach
kvclient/rangefeed: add rangefeed stuck watcher back to distsender rangefeed
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
We removed the rangefeed stuck watcher for mux rangefeed in https://github.com/cockroachdb/cockroach/pull/107268 since it didn't play well with mux rangefeeds. The rangefeed stuck watcher was a mechanism on the rangefeed client side that monitored for KV events, automatically restarting the rangefeed if no KV events were received for over a minute from kvserver. Now that we’ve fully transitioned to mux rangefeed and removed non-mux rangefeeds entirely, we’ve lost this feature entirely.
We should consider reintroducing it since I've found retry.stuck a very useful metrics during escalations, indicating issues coming from the kvserver rangefeed side. During escalations, CDC often see lots of rangefeed retry errors, and many of which are just due to normal KV operations like splits or merges. These make it hard to determine whether this issue originates in KV or CDC. From what I've observed in escalations, retry.stuck is the clearest indicator of a KV issue among all the retry errors, as it signals that client side hasn’t received events for over a minute, pointing more directly to a potential KV problem.
In addition, we could also consider including a new protocol where clients can send a rangefeed close request if it detects a stuck rangefeed. While the server-side implementation is done, the client-side support is not in place yet. (Old rangefeed guards mentioned this in https://cockroachlabs.slack.com/archives/C05JB0QURND/p1691008047037829)
Jira issue: CRDB-42518
Contributor guide
Assessment
This issue has not been assessed yet.