cockroachdb / cockroachdb/cockroach

bulkmerge: distributed merge permanently fails when a participating SQL instance goes down

Open
#167,491 3 comments 0 reactions 0 assignees View on GitHub
A-import A-schema-changes branch-release-26.2 C-enhancement docs-done docs-known-limitation O-agent T-sql-foundations
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary:**

When distributed merge is enabled, both IMPORT and index backfill permanently fail if a SQL instance that produced SSTs goes down and doesn't come back. This is a regression in resilience compared to the non-distributed merge path, which can redistribute work to surviving nodes.

**Background:**

Distributed merge writes intermediate SST files to `nodelocal` storage on each participating SQL instance. During the merge phase, the coordinator needs to read SSTs from all instances. If an instance becomes unavailable, `waitForRequiredInstances` in [`pkg/sql/bulkmerge/merge.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/merge.go) retries with exponential backoff for up to 30 minutes, then returns an `InstanceUnavailableError` marked as a permanent job error (no further retries).

In contrast, the non-distributed code path has no intermediate files tied to specific instances — it can simply re-distribute the remaining work to surviving nodes and complete successfully.

**Impact:**

- IMPORT jobs with `distmerge=true` fail permanently if a worker or coordinator node goes down during the merge phase.
- Index backfill jobs using distributed merge have the same limitation.
- The roachtest `import/nodeShutdown/{worker,coordinator}/distmerge=true` tests are affected (#167391, fix in #167445 skips those test variants).

**Code References:**

- [`pkg/sql/bulkmerge/merge.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/merge.go) — `waitForRequiredInstances`, 30-minute timeout
- [`pkg/sql/bulkmerge/instance_availability.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/bulkmerge/instance_availability.go) — `InstanceUnavailableError`, marked permanent
- [`pkg/sql/index_backfiller.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/index_backfiller.go) — backfill's call into `invokeBulkMerge`
- [`pkg/sql/importer/import_processor_planning.go`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/importer/import_processor_planning.go) — import's call into `bulkmerge.Merge`

Epic CRDB-62564
Jira issue: CRDB-62488

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.