Import data using Lightning from cloud storage failed due to "scan regions from start-key" error
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Import data using Lightning from cloud storage using Physical Import Mode (backend = local)
### 2. What did you expect to see? (Required)
Lightning succeeds.
### 3. What did you see instead (Required)
Lightning failed due to error from TiDB server.
```
tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region
[2024/01/11 08:42:09.464 +00:00] [ERROR] [main.go:103] [""tidb lightning encountered error stack info""] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tidb/br/pkg/lightning/common.NormalizeOrWrapErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/common/errors.go:222\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*Controller).importTables.func6\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/import.go:1656\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=] [engineUUID=317d5a82-b901-5f31-a8fb-9b4223031f4f] [retryCnt=0] [takeTime=1m26.668744789s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [import.go:2079] [""importing large engines for disk quota failed""] [diskSize=301382974674] [memSize=0] [quota=268435456000] [largeEnginesCount=1] [inProgressLargeEnginesCount=0] [takeTime=1m26.668819276s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:521] [""run failed""] [step=4] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:531] [""the whole procedure failed""] [takeTime=5h17m20.651341731s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:168] [""tables failed to be imported""] [count=1]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:170] [-] [table=`assets_proxy_prod_gl`.`assets`] [status=imported] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.680 +00:00] [ERROR] [import.go:1619] [""restore all tables data failed""] [takeTime=5h17m19.933853996s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1657] [""restore table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h17m19.668266409s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:1205] [""import and cleanup engine failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [takeTime=2m14.243945508s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:477] [""import whole table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h16m54.536504671s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1653] [""failed to import table""] [table=`assets_proxy_prod_gl`.`assets`] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[BR:PD:ErrPDBatchScanRegion]batch scan region\nscan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:57\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:75\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateJobForRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1218\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateAndSendJob.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1150\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:02.428 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [retryCnt=0] [takeTime=2m14.234574581s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
```
### 4. What is your TiDB version? (Required)
v7.1.2
Contributor guide
Research direction
Start by reproducing the v7.1.2 Physical Import Mode failure with Lightning using the described cloud-storage data and local backend. Read br/pkg/restore/split/split.go, especially PaginateScanRegion, and the local backend entry points in br/pkg/lightning/backend/local/local.go. Done means the import completes without the ErrPDBatchScanRegion context-canceled error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100