pingcap / pingcap/tidb

lightning parallel import "fetch tso from pd failed" failed with continuous injection of PD failures during import process

Open
#43,496 1 comment 0 reactions 1 assignee Claimed by @lichunzhu View on GitHub
component/lightning found/automation severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

two lightning parallel import 70G data
during import, continuous injection of PD failures

### 2. What did you expect to see? (Required)
lightning import successfully

### 3. What did you see instead (Required)
```code
2023/04/28 08:26:39.947 +00:00] [INFO] [table_restore.go:780] ["local checksum"] [table=`IFRS9`.`ECL_RETAIL_LOAN_BIZ_INFO`] [checksum="{cksum=11319510343246282912,size=104024366170,kvs=511748812}"]
[2023/04/28 08:26:39.947 +00:00] [INFO] [checksum.go:159] ["remote checksum start"] [table=ECL_RETAIL_LOAN_BIZ_INFO]
[2023/04/28 08:26:39.947 +00:00] [ERROR] [checksum.go:162] ["remote checksum failed"] [table=ECL_RETAIL_LOAN_BIZ_INFO] [takeTime=80.97µs] [error="fetch tso from pd failed: EOF"]
[2023/04/28 08:26:39.947 +00:00] [ERROR] [restore.go:1541] ["restore all tables data failed"] [takeTime=1h13m6.943873785s] [error="fetch tso from pd failed: EOF"]
[2023/04/28 08:26:39.947 +00:00] [ERROR] [restore.go:479] ["run failed"] [step=4] [error="fetch tso from pd failed: EOF"]
[2023/04/28 08:26:39.947 +00:00] [INFO] [restore.go:1184] ["everything imported, stopping periodic actions"]
[2023/04/28 08:26:39.947 +00:00] [ERROR] [restore.go:489] ["the whole procedure failed"] [takeTime=1h13m7.849785113s] [error="fetch tso from pd failed: EOF"]
[2023/04/28 08:26:39.979 +00:00] [ERROR] [main.go:103] ["tidb lightning encountered error stack info"] [error="fetch tso from pd failed: EOF"] [errorVerbose="EOF\ngithub.com/tikv/pd/client.(*client).processTSORequests\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1090\ngithub.com/tikv/pd/client.(*client).handleDispatcher\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:847\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\ngithub.com/tikv/pd/client.(*tsoRequest).Wait\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1313\ngithub.com/tikv/pd/client.(*client).GetTS\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20221031025758-80f0d8ca4d07/client.go:1333\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.(*tikvChecksumManager).Checksum\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/checksum.go:315\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.DoChecksum\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/checksum.go:161\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.(*TableRestore).postProcess\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/table_restore.go:845\ngithub.com/pingcap/tidb/br/pkg/lightning/restore.(*Controller).restoreTables.func7.1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/restore/restore.go:1668\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nfetch tso from pd failed"]
```
### 4. What is your TiDB version? (Required)

[2023/04/28 07:13:31.691 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v6.5.2-20230428] [git-hash=c4eccee42dba2a157d8ed945371e02ccf353457d] [git-branch=heads/refs/tags/v6.5.2-20230428] [go-version=go1.19.8] [utc-build-time="2023-04-28 02:32:46"] [race-enabled=false]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.