cockroachdb / cockroachdb/cockroach
bulkmerge: distributed merge permanently fails when a participating SQL instance goes down
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary:**
When distributed merge is enabled, both IMPORT and index backfill permanently fail if a SQL instance that produced SSTs goes down and doesn't come back. This is a regression in resilience compared to the non-distributed merge path, which can redistribute work to surviving nodes.
**Background:**
Distributed merge writes intermediate SST files to `nodelocal` storage on each participating SQL instance. During the merge phase, the coordinator needs to read SSTs from all instances. If an instance becomes unavailable, `waitForRequiredInstances` in [`pkg/sql/bulkmerge/merge.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/merge.go) retries with exponential backoff for up to 30 minutes, then returns an `InstanceUnavailableError` marked as a permanent job error (no further retries).
In contrast, the non-distributed code path has no intermediate files tied to specific instances — it can simply re-distribute the remaining work to surviving nodes and complete successfully.
**Impact:**
- IMPORT jobs with `distmerge=true` fail permanently if a worker or coordinator node goes down during the merge phase.
- Index backfill jobs using distributed merge have the same limitation.
- The roachtest `import/nodeShutdown/{worker,coordinator}/distmerge=true` tests are affected (#167391, fix in #167445 skips those test variants).
**Code References:**
- [`pkg/sql/bulkmerge/merge.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/merge.go) — `waitForRequiredInstances`, 30-minute timeout
- [`pkg/sql/bulkmerge/instance_availability.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/instance_availability.go) — `InstanceUnavailableError`, marked permanent
- [`pkg/sql/index_backfiller.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/index_backfiller.go) — backfill's call into `invokeBulkMerge`
- [`pkg/sql/importer/import_processor_planning.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/import_processor_planning.go) — import's call into `bulkmerge.Merge`
Epic CRDB-62564
Jira issue: CRDB-62488
Contributor guide
Assessment
This issue has not been assessed yet.