pingcap / pingcap/tidb

Br backup consumes mounts of time and gererates too many small files in million tables scenario

Open
#55,609 1 comment 0 reactions 0 assignees View on GitHub
affects-8.5 component/br feature/developing may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

In a scenario with a million databases and tables, `br full backup` constructs backup ranges based on the data and indexes of the tables. For example, with a million `sysbench` tables, it is necessary to construct 2 million ranges. `br` requests TiKV to upload the corresponding SST files to S3 according to the size of the ranges, which generates a large number of small files and the backup of the data is extremely time-consuming.

In actual testing, we created 1 million databases, 2 million tables, each table containing 2 row of data and one index, with approximately 50GB of for 3 replicas. The full backup took about 6 hours to reach a 60% progress, generating a million-level number of SST files. Ultimately, the backup task failed due to the creation of too many small files on Minio, causing Minio to return an error.
The minio files stats
```
[root@172 tiflash]# ls 1mdatabase-2table-2row-master6
1001 1004 1013 1019 1025 1026 backup.lock checkpoints
[root@172 tiflash]# ls 1mdatabase-2table-2row-master6/1013 | wc -l
525132
[root@172 tiflash]# ls 1mdatabase-2table-2row-master6/1019 | wc -l
868737
[root@172 tiflash]# ls 1mdatabase-2table-2row-master6/1025 | wc -l
1204584
```

### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)

### 4. What is your TiDB version? (Required)

master

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.