cockroachdb / cockroachdb/cockroach

kvclient/rangefeed: add rangefeed stuck watcher back to distsender rangefeed

Open
#131,356 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-rangefeed C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We removed the rangefeed stuck watcher for mux rangefeed in https://github.com/cockroachdb/cockroach/pull/107268 since it didn't play well with mux rangefeeds. The rangefeed stuck watcher was a mechanism on the rangefeed client side that monitored for KV events, automatically restarting the rangefeed if no KV events were received for over a minute from kvserver. Now that we’ve fully transitioned to mux rangefeed and removed non-mux rangefeeds entirely, we’ve lost this feature entirely.

We should consider reintroducing it since I've found retry.stuck a very useful metrics during escalations, indicating issues coming from the kvserver rangefeed side. During escalations, CDC often see lots of rangefeed retry errors, and many of which are just due to normal KV operations like splits or merges. These make it hard to determine whether this issue originates in KV or CDC. From what I've observed in escalations, retry.stuck is the clearest indicator of a KV issue among all the retry errors, as it signals that client side hasn’t received events for over a minute, pointing more directly to a potential KV problem.

In addition, we could also consider including a new protocol where clients can send a rangefeed close request if it detects a stuck rangefeed. While the server-side implementation is done, the client-side support is not in place yet. (Old rangefeed guards mentioned this in https://cockroachlabs.slack.com/archives/C05JB0QURND/p1691008047037829)

Jira issue: CRDB-42518

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.