pingcap / pingcap/tidb

Lots of SST files leaked if restore failed and resumed several times, and result in TiKV disk full.

Open
#50,633 1 comment 0 reactions 0 assignees View on GitHub
component/br severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1. Run "br restore full --tikv-max-restore-concurrency=256 xxx" to restore a 300TB cluster to a brancd new TiDB Cluster with 100 TiKV nodes, each TiKV with a 5TB disk attached.
2. br restore full was terminated due to br restore download SST throughtput exceeded GCP egress bandwidth quota of 1Tbps.
3. Tried to resume the br restore from checkpoint several times using different parameter --tikv-max-restore-concurrency=64 / 16 / 4, download SST throughtput still exceeded GCP quota
4. Finally restore can proceed without trigger GCP quota limit (download SST throughput ~640Gbps), when specifying parameters: --tikv-max-restore-concurrency=4 --ratelimit=800

### 2. What did you expect to see? (Required)
Final restore should succeed.

### 3. What did you see instead (Required)
Final restore failed due to TiKV disk full.

Before the final restore in step 4, each TiKV had about ~2TB of SST files leaked, as we could see from below screenshot.
![image](https://github.com/pingcap/tidb/assets/7403864/005efbc1-6ef8-4fb4-83c9-617145a301e8)

![image](https://github.com/pingcap/tidb/assets/7403864/595f12d1-574e-4141-b244-689ef23ab15a)

### 4. What is your TiDB version? (Required)
v7.6.0

Contributor guide

Open the contributing guide

Research direction

Start with the `br restore full` checkpoint-resume flow and the TiKV restore behavior described in the reproduction steps. Reproduce repeated interruptions with the listed concurrency and rate-limit parameters, then verify that resumed restores do not leave leaked SST files and can complete without filling TiKV disks.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.