[br] Pipeline region splitter can retain workers after SplitPoint fails
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
1. Run BR v8.5.8 with pipeline region splitting enabled for a log-restore or compacted-file restore.
2. Supply split metadata that causes at least one split worker to be submitted. Make that worker block in the region split/scatter operation.
3. Before the worker finishes, make a later `SplitPoint` operation fail, for example by returning a transient/exhausted PD region-scan error, a stale-region error, or a rewrite-key validation error.
4. Call `PipelineRegionsSplitterImpl.ExecuteRegions` and then attempt another split operation using the same process or worker pool.
The v8.5.8 runtime path was reproduced with a `SplitClient` test double: one split worker was submitted and blocked, then `SplitPoint` returned an error. `ExecuteRegions` returned without joining the in-flight worker or closing the regions channel.
### 2. What did you expect to see? (Required)
When `SplitPoint` fails, `ExecuteRegions` should cancel/join all submitted split workers, close the result channel only after all producers have stopped, and release the worker-pool resources before returning the error.
### 3. What did you see instead (Required)
`ExecuteRegions` returns immediately when `SplitPoint` returns an error. In that branch it does not wait for `r.eg`, close `regionsCh`, or wait for the scattering consumer. An already submitted split worker can therefore remain in flight and a worker can remain checked out from the pool after `ExecuteRegions` has returned.
This was directly reproduced in the v8.5.8 runtime path. Releasing the blocked worker allowed the pool to recover, which confirms that the leaked in-flight work is the resource-retention mechanism.
The failure input is reachable in real restore operations: PD region scans can fail or observe stale/changed regions during restore, and rewrite-key validation can fail while processing backup metadata. The expected user-visible result is a restore error; the additional defect is that cleanup is incomplete and later splitting/restore work in the same BR process can be delayed or blocked by retained workers.
Relevant code: `br/pkg/restore/split/splitter.go`, `PipelineRegionsSplitterImpl.ExecuteRegions`.
### 4. What is your TiDB version? (Required)
TiDB v8.5.8, commit `8b857efa20363d50a8fa2ea7dd9809a85a61b115`.
Related change: [PR #64850](https://github.com/pingcap/tidb/pull/64850), which introduced the pipeline splitter implementation.
Contributor guide
Research direction
Start in br/pkg/restore/split/splitter.go at PipelineRegionsSplitterImpl.ExecuteRegions, then reproduce the failure with the described SplitClient test double: block a submitted worker and make a later SplitPoint fail. Done means the error path cancels and joins submitted workers, closes regionsCh after producers stop, waits for the scattering consumer, and releases the worker-pool resources before returning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100