pingcap / pingcap/tidb

[br] Pipeline region splitter can retain workers after SplitPoint fails

Open
#70,806 2 comments 0 reactions 0 assignees View on GitHub
affects-25.10 affects-26.3 affects-8.5 component/br found-by-ai severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

1. Run BR v8.5.8 with pipeline region splitting enabled for a log-restore or compacted-file restore.
2. Supply split metadata that causes at least one split worker to be submitted. Make that worker block in the region split/scatter operation.
3. Before the worker finishes, make a later `SplitPoint` operation fail, for example by returning a transient/exhausted PD region-scan error, a stale-region error, or a rewrite-key validation error.
4. Call `PipelineRegionsSplitterImpl.ExecuteRegions` and then attempt another split operation using the same process or worker pool.

The v8.5.8 runtime path was reproduced with a `SplitClient` test double: one split worker was submitted and blocked, then `SplitPoint` returned an error. `ExecuteRegions` returned without joining the in-flight worker or closing the regions channel.

### 2. What did you expect to see? (Required)

When `SplitPoint` fails, `ExecuteRegions` should cancel/join all submitted split workers, close the result channel only after all producers have stopped, and release the worker-pool resources before returning the error.

### 3. What did you see instead (Required)

`ExecuteRegions` returns immediately when `SplitPoint` returns an error. In that branch it does not wait for `r.eg`, close `regionsCh`, or wait for the scattering consumer. An already submitted split worker can therefore remain in flight and a worker can remain checked out from the pool after `ExecuteRegions` has returned.

This was directly reproduced in the v8.5.8 runtime path. Releasing the blocked worker allowed the pool to recover, which confirms that the leaked in-flight work is the resource-retention mechanism.

The failure input is reachable in real restore operations: PD region scans can fail or observe stale/changed regions during restore, and rewrite-key validation can fail while processing backup metadata. The expected user-visible result is a restore error; the additional defect is that cleanup is incomplete and later splitting/restore work in the same BR process can be delayed or blocked by retained workers.

Relevant code: `br/pkg/restore/split/splitter.go`, `PipelineRegionsSplitterImpl.ExecuteRegions`.

### 4. What is your TiDB version? (Required)

TiDB v8.5.8, commit `8b857efa20363d50a8fa2ea7dd9809a85a61b115`.

Related change: [PR #64850](https://github.com/pingcap/tidb/pull/64850), which introduced the pipeline splitter implementation.

Contributor guide

Open the contributing guide

Research direction

Start in br/pkg/restore/split/splitter.go at PipelineRegionsSplitterImpl.ExecuteRegions, then reproduce the failure with the described SplitClient test double: block a submitted worker and make a later SplitPoint fail. Done means the error path cancels and joins submitted workers, closes regionsCh after producers stop, waits for the scattering consumer, and releases the worker-pool resources before returning.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.