cockroachdb / cockroachdb/cockroach

spanconfig: version-gated translator outputs never applied if the gate activates after the last full reconciliation

Open
#173,232 2 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution A-zone-configs C-bug O-agent O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

`SQLTranslator.generateSpanConfigurationsForNamedZone` emits different records for the meta named zone depending on a cluster version gate ([pkg/spanconfig/spanconfigsqltranslator/sqltranslator.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/spanconfig/spanconfigsqltranslator/sqltranslator.go)):

```go
case zonepb.MetaZoneName:
if s.settings.Version.IsActive(ctx, clusterversion.V26_1_InstallMeta2StaticSplitPoint) {
spans = append(spans, keys.Meta1Span)
spans = append(spans, roachpb.Span{Key: keys.Meta2Prefix, EndKey: keys.NodeLivenessSpan.Key})
} else {
spans = append(spans, roachpb.Span{Key: keys.Meta1Span.Key, EndKey: keys.NodeLivenessSpan.Key})
}
```

The record boundary at `Meta2Prefix` is the static split point from #155554 that allows meta2 to split independently of meta1 (#119421).

However, translator output is a function of `(SQL state, cluster version)`, and the reconciler only ever re-translates on changes to **SQL state**:

- The **full** pass (`fullReconciler.reconcile`) translates everything, but runs only when `Reconciler.Reconcile` starts — i.e. when the auto span config reconciliation job (re)starts ([pkg/spanconfig/spanconfigreconciler/reconciler.go](https://github.com/cockroachdb/cockroach/blob/master/pkg/spanconfig/spanconfigreconciler/reconciler.go)).
- The **incremental** pass re-translates only IDs surfaced by `SQLWatcher.WatchForSQLUpdates`, which is driven by rangefeeds over `system.descriptors` and `system.zones`. A cluster version bump writes to `system.settings`, which is not watched, so gate activation produces zero updates.

During a rolling upgrade the job coordinator moves as nodes restart, so full passes do happen — but all of them run *before* finalization, with the gate inactive. After finalization the gate is active but nothing restarts the job, so the new output is never computed. The cluster stays in the pre-gate state indefinitely, until a coincidental reconciler restart (node restart, job pause/resume, or the next upgrade).

**To Reproduce**

1. Run a cluster at a version predating the gate (e.g. v25.4).
2. Perform a rolling upgrade to 26.1+ and finalize; do not restart any node afterward.
3. Inspect `system.span_configurations`: the meta zone still has the single collapsed record spanning `/Min–/System/NodeLiveness`; no record boundary at `/Meta2` exists, and r1 remains a single range holding meta1+meta2.

**Expected behavior**

Version-gated span config outputs should be applied promptly once the gate activates — the meta2 split point should appear shortly after upgrade finalization without requiring a node restart or job pause/resume.

**Impact**

Observed in a production support escalation (https://github.com/cockroachlabs/support/issues/3671): the meta2 split point did not activate after finalizing to 26.2, leaving meta1+meta2 collapsed in r1. Heavy meta2 descriptor-scan load (changefeed range-stats pollers over a table with ~150k ranges) then pinned the r1 leaseholder at ~95% CPU, with the hotspot following the lease around the cluster; r1 could not load-split because it contained meta1 (#119421). Mitigated with a manual `AdminSplit` per runbook.

**Proposed fixes**

1. *(general)* Trigger reconciliation on cluster version changes: register a callback on version-setting changes (or compare the active version at checkpoint time) and restart `Reconcile` — whose initial full pass recomputes all translator outputs — when it advances. Version changes are rare, so the extra full pass is cheap. Note the interaction with the existing TODO in `Reconciler.Reconcile` about *avoiding* full passes on job restart: if that lands, restarts stop papering over this bug and a version-change trigger becomes required for correctness.
2. *(point fix, backportable)* Add an upgrade migration ordered after `V26_1_InstallMeta2StaticSplitPoint` that touches the meta named zone's `system.zones` row, generating a rangefeed event so the incremental reconciler re-translates it with the gate active. This also heals clusters already stuck in the collapsed state on their next upgrade.

**Environment:**
- Observed on v26.2.4 (gate `V26_1_InstallMeta2StaticSplitPoint`); mechanism unchanged on master.

**Additional context**

Related: #155554 (introduced the gated split point), #119421 (meta1 prevents meta2 load splitting). This is a class bug: any future version-gated translator output has the same missing invalidation trigger.

Jira issue: CRDB-66547

Contributor guide

Open the contributing guide

Research direction

Read pkg/spanconfig/spanconfigsqltranslator/sqltranslator.go and pkg/spanconfig/spanconfigreconciler/reconciler.go, then trace how SQLWatcher.WatchForSQLUpdates triggers incremental reconciliation. Reproduce the upgrade-finalization scenario and verify that the version-gated meta zone output is recomputed after activation, producing a boundary at /Meta2 without a node restart or job pause.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.