pingcap / pingcap/tidb

Import data using Lightning from cloud storage failed due to "scan regions from start-key" error

Open
#50,333 3 comments 0 reactions 0 assignees View on GitHub
affects-7.1 component/lightning severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Import data using Lightning from cloud storage using Physical Import Mode (backend = local)

### 2. What did you expect to see? (Required)

Lightning succeeds.

### 3. What did you see instead (Required)

Lightning failed due to error from TiDB server.

```
tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region
[2024/01/11 08:42:09.464 +00:00] [ERROR] [main.go:103] [""tidb lightning encountered error stack info""] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tidb/br/pkg/lightning/common.NormalizeOrWrapErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/common/errors.go:222\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*Controller).importTables.func6\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/import.go:1656\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=] [engineUUID=317d5a82-b901-5f31-a8fb-9b4223031f4f] [retryCnt=0] [takeTime=1m26.668744789s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [import.go:2079] [""importing large engines for disk quota failed""] [diskSize=301382974674] [memSize=0] [quota=268435456000] [largeEnginesCount=1] [inProgressLargeEnginesCount=0] [takeTime=1m26.668819276s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:521] [""run failed""] [step=4] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:531] [""the whole procedure failed""] [takeTime=5h17m20.651341731s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:168] [""tables failed to be imported""] [count=1]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:170] [-] [table=`assets_proxy_prod_gl`.`assets`] [status=imported] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.680 +00:00] [ERROR] [import.go:1619] [""restore all tables data failed""] [takeTime=5h17m19.933853996s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1657] [""restore table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h17m19.668266409s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:1205] [""import and cleanup engine failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [takeTime=2m14.243945508s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:477] [""import whole table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h16m54.536504671s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1653] [""failed to import table""] [table=`assets_proxy_prod_gl`.`assets`] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[BR:PD:ErrPDBatchScanRegion]batch scan region\nscan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:57\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:75\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateJobForRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1218\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateAndSendJob.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1150\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:02.428 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [retryCnt=0] [takeTime=2m14.234574581s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
```

### 4. What is your TiDB version? (Required)

v7.1.2

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the v7.1.2 Physical Import Mode failure with Lightning using the described cloud-storage data and local backend. Read br/pkg/restore/split/split.go, especially PaginateScanRegion, and the local backend entry points in br/pkg/lightning/backend/local/local.go. Done means the import completes without the ErrPDBatchScanRegion context-canceled error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.