cockroachdb / cockroachdb/cockroach
bulk: unhealthy sql instances hot spots bulk coordinator
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The bulk oracle (`bulkOracle.ChoosePreferredReplica`, `pkg/kv/followerreads/followerreads.go`) chooses a **KV replica node** from the range's replica set — applying locality filters and streak balancing, but with **no visibility into SQL instance health**. The healthy-SQL-instance set is never passed to the oracle; unlike the closest/bin-packing oracles it doesn't even consult the KV-level `HealthFunc`.
SQL instance health is applied separately and downstream, in `makeInstanceResolver` (`pkg/sql/distsql_physical_planner.go`), which maps the oracle's chosen KV node to the *closest healthy SQL instance* via `filterUnhealthyInstances` + `ClosestInstances`.
Because these two stages don't share the health set, the oracle can commit to a node whose only nearby healthy SQL instance is the gateway. When `ClosestInstances` finds no instance sharing locality tiers with the chosen node — or the closest one is the gateway — the resolver falls back to the gateway. `filterUnhealthyInstances` treats the gateway (the coordinator) as always healthy, so misrouted spans pile onto the coordinator instead of being distributed.
The root issue is the decoupling: node selection is health-unaware, and the health-aware instance mapping happens too late to influence which node was chosen.
**Impact**
Work concentrates on the coordinator when the oracle picks nodes that don't have a co-located healthy SQL instance. All operations that use the bulk oracle are affected:
| Job / operation | Constructor | Gating |
|---|---|---|
| Changefeeds | `NewLocalityFilteringBulkOracle` | `changefeed.random_replica_selection.enabled`, default on |
| Backup | `NewLocalityFilteringBulkOracle` | `bulkio.backup.balanced_distribution.enabled`, default on |
| Backup compaction | `NewLocalityFilteringBulkOracle` | same `bulkio.backup.balanced_distribution.enabled` setting |
| Cross-cluster replication (PCR/LDR producer) | `NewStreakBulkOracle` | via `useStreaks` |
| Fingerprint (`SHOW EXPERIMENTAL_FINGERPRINTS`) | `NewStreakBulkOracle` | always |
**Workaround (changefeeds only)**
Setting `changefeed.default_range_distribution_strategy = 'balanced_simple'` redistributes ranges evenly across selected nodes after the fact, avoiding the pile-up on the coordinator. This setting is changefeed-specific — backup, backup compaction, cross-cluster replication, and fingerprint have no equivalent escape hatch.
**Code references**
- [followerreads.go — `ChoosePreferredReplica`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/followerreads/followerreads.go#L233)
- [distsql_physical_planner.go — `makeInstanceResolver`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/distsql_physical_planner.go#L1385)
- [distsql_physical_planner.go — `filterUnhealthyInstances`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/distsql_physical_planner.go#L1363)
**Fix direction**
Health should inform node selection, not just the post-hoc instance mapping — e.g. make the oracle aware of which KV nodes have a healthy co-located SQL instance, or filter the candidate nodes to those with a healthy instance before the oracle commits. See the existing `TODO(dt)` at `followerreads.go:259` and the unify-oracles TODO (#120755).
Contributor guide
Research direction
Read ChoosePreferredReplica in pkg/kv/followerreads/followerreads.go, then trace makeInstanceResolver and filterUnhealthyInstances in pkg/sql/distsql_physical_planner.go. Review the TODO at followerreads.go:259 and the unify-oracles TODO (#120755) before deciding how health should reach node selection. Done means bulk-oracle choices account for healthy co-located SQL instances and no longer funnel spans to the gateway when a suitable instance exists.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100