cockroachdb / cockroachdb/cockroach

bulk: unhealthy sql instances hot spots bulk coordinator

Open
#173,077 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-bug O-agent T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

The bulk oracle (`bulkOracle.ChoosePreferredReplica`, `pkg/kv/followerreads/followerreads.go`) chooses a **KV replica node** from the range's replica set — applying locality filters and streak balancing, but with **no visibility into SQL instance health**. The healthy-SQL-instance set is never passed to the oracle; unlike the closest/bin-packing oracles it doesn't even consult the KV-level `HealthFunc`.

SQL instance health is applied separately and downstream, in `makeInstanceResolver` (`pkg/sql/distsql_physical_planner.go`), which maps the oracle's chosen KV node to the *closest healthy SQL instance* via `filterUnhealthyInstances` + `ClosestInstances`.

Because these two stages don't share the health set, the oracle can commit to a node whose only nearby healthy SQL instance is the gateway. When `ClosestInstances` finds no instance sharing locality tiers with the chosen node — or the closest one is the gateway — the resolver falls back to the gateway. `filterUnhealthyInstances` treats the gateway (the coordinator) as always healthy, so misrouted spans pile onto the coordinator instead of being distributed.

The root issue is the decoupling: node selection is health-unaware, and the health-aware instance mapping happens too late to influence which node was chosen.

**Impact**

Work concentrates on the coordinator when the oracle picks nodes that don't have a co-located healthy SQL instance. All operations that use the bulk oracle are affected:

| Job / operation | Constructor | Gating |
|---|---|---|
| Changefeeds | `NewLocalityFilteringBulkOracle` | `changefeed.random_replica_selection.enabled`, default on |
| Backup | `NewLocalityFilteringBulkOracle` | `bulkio.backup.balanced_distribution.enabled`, default on |
| Backup compaction | `NewLocalityFilteringBulkOracle` | same `bulkio.backup.balanced_distribution.enabled` setting |
| Cross-cluster replication (PCR/LDR producer) | `NewStreakBulkOracle` | via `useStreaks` |
| Fingerprint (`SHOW EXPERIMENTAL_FINGERPRINTS`) | `NewStreakBulkOracle` | always |

**Workaround (changefeeds only)**

Setting `changefeed.default_range_distribution_strategy = 'balanced_simple'` redistributes ranges evenly across selected nodes after the fact, avoiding the pile-up on the coordinator. This setting is changefeed-specific — backup, backup compaction, cross-cluster replication, and fingerprint have no equivalent escape hatch.

**Code references**

- [followerreads.go — `ChoosePreferredReplica`](https://github.com/cockroachdb/cockroach/blob/master/pkg/kv/followerreads/followerreads.go#L233)
- [distsql_physical_planner.go — `makeInstanceResolver`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/distsql_physical_planner.go#L1385)
- [distsql_physical_planner.go — `filterUnhealthyInstances`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/distsql_physical_planner.go#L1363)

**Fix direction**

Health should inform node selection, not just the post-hoc instance mapping — e.g. make the oracle aware of which KV nodes have a healthy co-located SQL instance, or filter the candidate nodes to those with a healthy instance before the oracle commits. See the existing `TODO(dt)` at `followerreads.go:259` and the unify-oracles TODO (#120755).

Contributor guide

Open the contributing guide

Research direction

Read ChoosePreferredReplica in pkg/kv/followerreads/followerreads.go, then trace makeInstanceResolver and filterUnhealthyInstances in pkg/sql/distsql_physical_planner.go. Review the TODO at followerreads.go:259 and the unify-oracles TODO (#120755) before deciding how health should reach node selection. Done means bulk-oracle choices account for healthy co-located SQL instances and no longer funnel spans to the gateway when a suitable instance exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.