cockroachdb / cockroachdb/cockroach
sqlliveness: sql instance watcher should handle rangefeed errors
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The [sql instance watcher](https://github.com/cockroachdb/cockroach/blob/046fd13d8fa500ff4389bf0d3402374c498b6303/pkg/sql/sqlinstance/instancestorage/instancereader.go#L157) relies on rangefeeds to update its internal cache of live sql instances. Rangefeed clients based on `kvclient/rangefeed.Factory` must explicitly set an `OnInternalError` handler to be notified of internal errors. Otherwise, when one of these errors are encountered the rangefeed will silently stop emitting events. This can cause the sql instance cache to get stale leading to the system believe certain instances are alive and accessible after they have already been shut down.
This issue tracks the work to update the sql instance watcher to ensure we handle such errors gracefully and prevent the cache from getting stale. This is a subset of the work identified in https://github.com/cockroachdb/cockroach/issues/75610.
Jira issue: CRDB-49869
Epic CRDB-52349
Contributor guide
Research direction
Start with the sql instance watcher in pkg/sql/sqlinstance/instancestorage/instancereader.go at the rangefeed setup, then read the kvclient/rangefeed.Factory contract for OnInternalError. Trace how internal rangefeed errors affect the watcher and its cache; done means those errors are handled gracefully so the cache cannot remain stale after instances shut down.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100