pingcap / pingcap/tidb

Pitr restore speed is too slow for many split files in million tables scenario

Open
#58,002 2 comments 0 reactions 0 assignees View on GitHub
affects-8.5 component/br may-affects-5.4 may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 severity/major type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1 million tables and read/write on 100k tables
log backcup
log restore to a empty cluster , it takes 16h11m to restore for 171.5GB data.
```
[2024/12/04 18:53:05.417 +00:00] [INFO] [collector.go:77] ["restore log success summary"] [total-take=16h11m42.662016972s] [source-start-point=454357264218456070] [source-end-point=454366569701310464] [target-end-point=454368974014251042] [source-start="2024-12-03 14:16:53.431 +0000"] [source-end="2024-12-04 00:08:31.031 +0000"] [target-end="2024-12-04 02:41:22.757 +0000"] [total-kv-count=1111389936] [skipped-kv-count-by-checkpoint=0] [total-size=171.5GB] [skipped-size-by-checkpoint=0B] [average-speed=2.942MB/s]
```
The br nearly need process 3048431 * 8 files when log restore.
```
[2024/12/04 17:50:03.865 +00:00] [INFO] [client.go:754] ["import files done"] [batch-count=8] [batch-size=65776] [take=578.003479ms] [files="[\"v1/20241203/23/1014/454365904484696483-f35f3fe6-ff04-46d3-a30e-5386f782f935.log, \",\"v1/20241203/23/1014/454365968185688965-92041c8d-bae3-4849-b228-fac3e6ae1d4f.log, \",\"v1/20241203/23/1014/454366008831116054-05df5a89-4a85-4b81-9732-79c9da194001.log, \",\"v1/20241203/23/1014/454366008870437639-fb7a8f5a-4643-48a1-aee0-98cbf63f4d26.log, \",\"v1/20241203/23/1014/454366090121969819-13a6713c-d02f-4e76-8ec8-0feb7e6a7741.log, \",\"v1/20241203/23/1014/454366101132018218-419493d0-1ac4-44ff-84e5-561631b322e9.log, \",\"v1/20241203/23/1014/454366171399716920-c9b96085-4689-40c5-833b-e433c4be914b.log, \",\"v1/20241203/23/1014/454366210734424257-2ee79d68-9306-4a22-aae7-6bd198c4c5ec.log, \"]"]
```
```
root@benchtoolset2-0:/# grep "import files done" /tmp/br.log.2024-12-04T00.14* | wc -l
3048431
```
when increase the concurrency the restore speed increase(5h38m) but the restore process will pause for long time for tikv flow control
```
[2024/12/04 23:56:43.972 +00:00] [INFO] [collector.go:77] ["restore log success summary"] [total-take=5h38m50.532165537s] [source-start-point=454357264218456070] [source-end-point=454366569701310464] [target-end-point=454383704066752552] [source-start="2024-12-03 14:16:53.431 +0000"] [source-end="2024-12-04 00:08:31.031 +0000"] [target-end="2024-12-04 18:17:53.448 +0000"] [total-kv-count=1111389936] [skipped-kv-count-by-checkpoint=0] [total-size=171.5GB] [skipped-size-by-checkpoint=0B] [average-speed=8.437MB/s]
```
![Image](https://github.com/user-attachments/assets/052ee002-a3d6-4003-bee4-1ce0c6c9cbf5)
![Image](https://github.com/user-attachments/assets/525baea0-6922-41d1-acfe-b42993ac086c)
the br log in the paused time
```
[2024/12/04 18:47:47.636 +00:00] [INFO] [store_cache.go:1060] ["store health status changed"] [storeID=1001] [isSlow=false] [healthDetail="{ ClientSideSlowScore: 1, TiKVSideSlowScore: 70 }"]
[2024/12/04 18:47:47.952 +00:00] [INFO] [store_cache.go:1060] ["store health status changed"] [storeID=1001] [isSlow=true] [healthDetail="{ ClientSideSlowScore: 100, TiKVSideSlowScore: 70 }"]
[2024/12/04 18:47:53.494 +00:00] [INFO] [pd.go:329] ["pause scheduler(configs)"] [name="[balance-region-scheduler,balance-leader-scheduler,balance-hot-region-scheduler]"] [cfg="{\"enable-location-replacement\":\"false\",\"leader-schedule-limit\":24,\"max-pending-peer-count\":2147483647,\"max-snapshot-count\":40,\"merge-schedule-limit\":0,\"region-schedule-limit\":40}"]
[2024/12/04 18:48:02.635 +00:00] [INFO] [store_cache.go:1060] ["store health status changed"] [storeID=1001] [isSlow=false] [healthDetail="{ ClientSideSlowScore: 1, TiKVSideSlowScore: 70 }"]
[2024/12/04 18:48:13.191 +00:00] [INFO] [store_cache.go:1060] ["store health status changed"] [storeID=1001] [isSlow=true] [healthDetail="{ ClientSideSlowScore: 100, TiKVSideSlowScore: 62 }"]
[2024/12/04 18:48:14.951 +00:00] [INFO] [progress.go:176] [progress] [step="Restore Files(SST + KV)"] [progress=6.81%] [count="83473512 / 1225755334"] [speed="206435 p/s"] [elapsed=20m0s] [remaining=1h32m13s]
```
### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)

### 4. What is your TiDB version? (Required)

Release Version: v8.5.0-alpha-279-g9812d85d0d
Git Commit Hash: 9812d85d0d259547cf1dae88abbc7c406c56f935
Git Branch: HEAD
Go Version: go1.23.3
UTC Build Time: 2024-12-03 17:06:19
Race Enabled: false

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.