cockroachdb / cockroachdb/cockroach

sql/schemachanger: slow rollout of distributed merge in Cloud via RAFA

Open
#167,409 1 comment 0 reactions 1 assignee Claimed by @spilchen View on GitHub
A-schema-changer-impl C-enhancement O-agent O-premortem T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**
Use RAFA to do a slow rollout of the `bulkio.index_backfill.distributed_merge.mode` cluster setting on Cloud clusters during the 26.3 dev cycle. This will give us real-world signal on whether the feature works correctly before switching it on by default.

From the 26.2 pre-mortem.

**Cluster setting:** `bulkio.index_backfill.distributed_merge.mode`
- Current default: `disabled`
- Target value for rollout: `enabled`
- Defined in `pkg/sql/backfill/distributed_merge_mode.go`

**Precedent:**
We followed a similar process for `sql.defaults.autocommit_before_ddl.enabled` in https://github.com/cockroachdb/cockroach/issues/133180 and https://github.com/cockroachlabs/rafa-production/pull/512.

**Rollout plan:**
1. Wait for a 26.2 patch release to be deployed in CockroachDB Cloud.
2. Ensure the SLI dashboard charts are in place (see related issue for dashboard work).
3. Create a RAFA PR to set `bulkio.index_backfill.distributed_merge.mode = 'enabled'` on an initial set of high-risk clusters.
4. Monitor the SLI dashboard and Datadog logs for the setting change.
5. If no issues after ~1 week, expand to a broader set of clusters.
6. Once confident, change the hard-coded default from `disabled` to `enabled` in `pkg/sql/backfill/distributed_merge_mode.go`.

**Next Steps:**
- [ ] Create the RAFA PR targeting an initial cluster set
- [ ] Monitor for regressions after each rollout stage
- [ ] Change the hard-coded default once rollout is validated

Jira issue: CRDB-62441

Epic CRDB-62564

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.