cockroachdb / cockroachdb/cockroach

sqlliveness: sql instance watcher should handle rangefeed errors

Open
#145,356 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-postmortem P-3 T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The [sql instance watcher](https://github.com/cockroachdb/cockroach/blob/046fd13d8fa500ff4389bf0d3402374c498b6303/pkg/sql/sqlinstance/instancestorage/instancereader.go#L157) relies on rangefeeds to update its internal cache of live sql instances. Rangefeed clients based on `kvclient/rangefeed.Factory` must explicitly set an `OnInternalError` handler to be notified of internal errors. Otherwise, when one of these errors are encountered the rangefeed will silently stop emitting events. This can cause the sql instance cache to get stale leading to the system believe certain instances are alive and accessible after they have already been shut down.

This issue tracks the work to update the sql instance watcher to ensure we handle such errors gracefully and prevent the cache from getting stale. This is a subset of the work identified in https://github.com/cockroachdb/cockroach/issues/75610.

Jira issue: CRDB-49869

Epic CRDB-52349

Contributor guide

Open the contributing guide

Research direction

Start with the sql instance watcher in pkg/sql/sqlinstance/instancestorage/instancereader.go at the rangefeed setup, then read the kvclient/rangefeed.Factory contract for OnInternalError. Trace how internal rangefeed errors affect the watcher and its cache; done means those errors are handled gracefully so the cache cannot remain stale after instances shut down.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.