Lots of SST files leaked if restore failed and resumed several times, and result in TiKV disk full.
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. Run "br restore full --tikv-max-restore-concurrency=256 xxx" to restore a 300TB cluster to a brancd new TiDB Cluster with 100 TiKV nodes, each TiKV with a 5TB disk attached.
2. br restore full was terminated due to br restore download SST throughtput exceeded GCP egress bandwidth quota of 1Tbps.
3. Tried to resume the br restore from checkpoint several times using different parameter --tikv-max-restore-concurrency=64 / 16 / 4, download SST throughtput still exceeded GCP quota
4. Finally restore can proceed without trigger GCP quota limit (download SST throughput ~640Gbps), when specifying parameters: --tikv-max-restore-concurrency=4 --ratelimit=800
### 2. What did you expect to see? (Required)
Final restore should succeed.
### 3. What did you see instead (Required)
Final restore failed due to TiKV disk full.
Before the final restore in step 4, each TiKV had about ~2TB of SST files leaked, as we could see from below screenshot.


### 4. What is your TiDB version? (Required)
v7.6.0
Contributor guide
Research direction
Start with the `br restore full` checkpoint-resume flow and the TiKV restore behavior described in the reproduction steps. Reproduce repeated interruptions with the listed concurrency and rate-limit parameters, then verify that resumed restores do not leave leaked SST files and can complete without filling TiKV disks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100