cockroachdb / cockroachdb/cockroach
kvclient/rangefeed: bounded retry options cause silent permanent feed death
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
If a caller supplies `retry.Options` with `MaxRetries` or `MaxDuration` via `WithRetry`, exhaustion of the retry loop makes both `RangeFeed.run` ([rangefeed.go#L360](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/rangefeed.go#L360), [#L429](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/rangefeed.go#L429)) and `runInitialScan` ([scanner.go#L92](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/scanner.go#L92), [#L124-L125](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/scanner.go#L124-L125)) exit silently: no `onUnrecoverableError`, no `OnInitialScanError`, no log. The feed permanently stops and the caller has no way to notice.
No production caller currently sets a bound, but nothing prevents it, and many consumers of this package (settings watchers, stats cache, spanconfig) treat a running feed as a liveness invariant — a silently dead feed there means silently stale data.
**Suggested fix**
Invoke `onUnrecoverableError` (or at minimum log loudly) when the retry loop is exhausted rather than context-canceled. Alternatively, reject bounded retry options in `WithRetry` since the component's consumers assume run-forever semantics.
**Additional context**
Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.
Jira issue: CRDB-65660
Contributor guide
Research direction
Read the cited retry-loop paths in pkg/kv/kvclient/rangefeed/rangefeed.go and scanner.go, starting at RangeFeed.run and runInitialScan, and trace how bounded exhaustion differs from context cancellation. Done means an exhausted retry loop is observable through the appropriate error callback or loud logging rather than silently stopping the feed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100