cockroachdb / cockroachdb/cockroach

kvclient/rangefeed: bounded retry options cause silent permanent feed death

Open
#172,424 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed branch-master C-bug O-agent P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

If a caller supplies `retry.Options` with `MaxRetries` or `MaxDuration` via `WithRetry`, exhaustion of the retry loop makes both `RangeFeed.run` ([rangefeed.go#L360](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/rangefeed.go#L360), [#L429](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/rangefeed.go#L429)) and `runInitialScan` ([scanner.go#L92](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/scanner.go#L92), [#L124-L125](https://github.com/cockroachdb/cockroach/blob/a7e117882b4356096b3a8ea26046baa7e5d53fce/pkg/kv/kvclient/rangefeed/scanner.go#L124-L125)) exit silently: no `onUnrecoverableError`, no `OnInitialScanError`, no log. The feed permanently stops and the caller has no way to notice.

No production caller currently sets a bound, but nothing prevents it, and many consumers of this package (settings watchers, stats cache, spanconfig) treat a running feed as a liveness invariant — a silently dead feed there means silently stale data.

**Suggested fix**

Invoke `onUnrecoverableError` (or at minimum log loudly) when the retry loop is exhausted rather than context-canceled. Alternatively, reject bounded retry options in `WithRetry` since the component's consumers assume run-forever semantics.

**Additional context**

Found during an agent-assisted correctness audit of the rangefeed subsystem. Code links are pinned to master @ a7e1178.

Jira issue: CRDB-65660

Contributor guide

Open the contributing guide

Research direction

Read the cited retry-loop paths in pkg/kv/kvclient/rangefeed/rangefeed.go and scanner.go, starting at RangeFeed.run and runInitialScan, and trace how bounded exhaustion differs from context cancellation. Done means an exhausted retry loop is observable through the appropriate error callback or loud logging rather than silently stopping the feed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.