lightning engines are not pipelined, what's the difference of using 6 engines or 1 engine?
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
From metrics we can see importing stage are not pipelined

(⬆️ in this metrics, index engine is started importing at ~20:50)
and from log, we can see 6 engines (which is equal to table-concurrency) enter encoding stage and importing stage almost at the same time
```
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=53] [takeTime=43m3.821352568s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=52] [takeTime=43m3.821620229s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:53] [engineUUID=78a5e84a-05ce-5885-a0d8-bc94633a01da]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:52] [engineUUID=e0d56bf7-9fd8-5a68-a13e-358674a81391]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:53] [engineUUID=78a5e84a-05ce-5885-a0d8-bc94633a01da] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:52] [engineUUID=e0d56bf7-9fd8-5a68-a13e-358674a81391] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=48] [takeTime=43m3.821901788s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=51] [takeTime=43m3.821887337s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=49] [takeTime=43m3.821516891s] []
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:51] [engineUUID=a98f9d7d-eb2b-53da-aba8-aac887d009fc]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:49] [engineUUID=084e3e0b-2a38-5894-b64d-3bea91a6b771]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:51] [engineUUID=a98f9d7d-eb2b-53da-aba8-aac887d009fc] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:49] [engineUUID=084e3e0b-2a38-5894-b64d-3bea91a6b771] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:443] ["restore engine start"] [table=`db`.`tbl`] [engineNumber=54]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:576] ["encode kv data and write start"] [table=`db`.`tbl`] [engineNumber=54]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:48] [engineUUID=95bccbb8-3376-5ea9-9faf-56fc8b8b034d]
[2023/05/10 03:23:03.657 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:48] [engineUUID=95bccbb8-3376-5ea9-9faf-56fc8b8b034d] [retryCnt=0]
[2023/05/10 03:23:03.657 +00:00] [INFO] [table_import.go:445] ["restore engine completed"] [table=`db`.`tbl`] [engineNumber=50] [takeTime=43m3.822173127s] []
[2023/05/10 03:23:03.658 +00:00] [INFO] [table_import.go:1138] ["import and cleanup engine start"] [engineTag=`db`.`tbl`:50] [engineUUID=5fa5d08a-0e2c-5590-a6d8-7b42c868c9dc]
[2023/05/10 03:23:03.658 +00:00] [INFO] [backend.go:488] ["import start"] [engineTag=`db`.`tbl`:50] [engineUUID=5fa5d08a-0e2c-5590-a6d8-7b42c868c9dc] [retryCnt=0]
```
Contributor guide
Assessment
This issue has not been assessed yet.