pingcap / pingcap/tidb

lightning engines are not pipelined, what's the difference of using 6 engines or 1 engine?

Open
#43,750 3 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement
From metrics we can see importing stage are not pipelined

![img_v2_29436c9d-d40a-4f7b-9944-b05387b7d71g](https://github.com/pingcap/tidb/assets/1689766/00ee5b76-2c5d-4aa9-8ef3-15a79cd9ebca)

(⬆️ in this metrics, index engine is started importing at ~20:50)

and from log, we can see 6 engines (which is equal to table-concurrency) enter encoding stage and importing stage almost at the same time

```
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=53] [takeTime=43m3.821352568s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=52] [takeTime=43m3.821620229s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:53] [engineUUID=78a5e84a-05ce-5885-a0d8-bc94633a01da]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:52] [engineUUID=e0d56bf7-9fd8-5a68-a13e-358674a81391]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:53] [engineUUID=78a5e84a-05ce-5885-a0d8-bc94633a01da] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:52] [engineUUID=e0d56bf7-9fd8-5a68-a13e-358674a81391] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=48] [takeTime=43m3.821901788s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=51] [takeTime=43m3.821887337s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=49] [takeTime=43m3.821516891s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:51] [engineUUID=a98f9d7d-eb2b-53da-aba8-aac887d009fc]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:49] [engineUUID=084e3e0b-2a38-5894-b64d-3bea91a6b771]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:51] [engineUUID=a98f9d7d-eb2b-53da-aba8-aac887d009fc] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:49] [engineUUID=084e3e0b-2a38-5894-b64d-3bea91a6b771] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:443] ["restore engine start"] [table=`db`.`tbl`] [engineNumber=54]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:576] ["encode kv data and write start"] [table=`db`.`tbl`] [engineNumber=54]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:48] [engineUUID=95bccbb8-3376-5ea9-9faf-56fc8b8b034d]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:48] [engineUUID=95bccbb8-3376-5ea9-9faf-56fc8b8b034d] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=50] [takeTime=43m3.822173127s] []
[2023/05/10 03:23:03.658 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:50] [engineUUID=5fa5d08a-0e2c-5590-a6d8-7b42c868c9dc]
[2023/05/10 03:23:03.658 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:50] [engineUUID=5fa5d08a-0e2c-5590-a6d8-7b42c868c9dc] [retryCnt=0]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.