cockroachdb / cockroachdb/cockroach
sql/schemachanger: slow rollout of distributed merge in Cloud via RAFA
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary:**
Use RAFA to do a slow rollout of the `bulkio.index_backfill.distributed_merge.mode` cluster setting on Cloud clusters during the 26.3 dev cycle. This will give us real-world signal on whether the feature works correctly before switching it on by default.
From the 26.2 pre-mortem.
**Cluster setting:** `bulkio.index_backfill.distributed_merge.mode`
- Current default: `disabled`
- Target value for rollout: `enabled`
- Defined in `pkg/sql/backfill/distributed_merge_mode.go`
**Precedent:**
We followed a similar process for `sql.defaults.autocommit_before_ddl.enabled` in https://github.com/cockroachdb/cockroach/issues/133180 and https://github.com/cockroachlabs/rafa-production/pull/512.
**Rollout plan:**
1. Wait for a 26.2 patch release to be deployed in CockroachDB Cloud.
2. Ensure the SLI dashboard charts are in place (see related issue for dashboard work).
3. Create a RAFA PR to set `bulkio.index_backfill.distributed_merge.mode = 'enabled'` on an initial set of high-risk clusters.
4. Monitor the SLI dashboard and Datadog logs for the setting change.
5. If no issues after ~1 week, expand to a broader set of clusters.
6. Once confident, change the hard-coded default from `disabled` to `enabled` in `pkg/sql/backfill/distributed_merge_mode.go`.
**Next Steps:**
- [ ] Create the RAFA PR targeting an initial cluster set
- [ ] Monitor for regressions after each rollout stage
- [ ] Change the hard-coded default once rollout is validated
Jira issue: CRDB-62441
Epic CRDB-62564
Contributor guide
Assessment
This issue has not been assessed yet.