pingcap / pingcap/tidb

Lightning stuck after tikv is down

Open
#33,864 6 comments 0 reactions 0 assignees View on GitHub
component/lightning
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

1. dump table using dumpling
2. restore table with lightning: `tidb-lightning --config ./lightning.toml`
3. tikv crashed for some reasons
4. lightning got stuck

lightning log:
```
Verbose debug logs will be written to /tmp/tidb-lightning.log

+---+----------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+---+----------------------------------------------+-------------+--------+
| 1 | Source csv files size is proper | performance | true |
+---+----------------------------------------------+-------------+--------+
| 2 | checkpoints are valid | critical | true |
+---+----------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+---+----------------------------------------------+-------------+--------+
| 4 | Cluster is available | critical | true |
+---+----------------------------------------------+-------------+--------+
| 5 | Lightning has the correct storage permission | critical | true |
+---+----------------------------------------------+-------------+--------+
[2022/04/11 08:10:02.441 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40052->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.441 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40050->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40048->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40046->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40038->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40042->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40036->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40040->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40030->10.244.5.105:20160: write: broken pipe"] [system=grpc] [grpc_log=true]
```

lightning.toml
```
[lightning]
# Logging
file = "/tmp/tidb-lightning.log"
level = "info"

[tikv-importer]
# Uses the Local-backend
backend = "local"
# Sets the directory for temporarily storing the sorted key-value pairs.
# The target directory must be empty.
sorted-kv-dir = "/tmp/sst"

[mydumper]
# Local source data directory
data-source-dir = "/tmp/dumpling"

# Configures the wildcard rule. By default, all tables in the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, and INSPECTION_SCHEMA system databases are filtered.
# If this item is not configured, the "cannot find schema" error occurs when system tables are imported.
filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*']
[tidb]
# Information of the target cluster
host = "172.16.102.67"
port = 4000
user = "root"
# Table schema information is fetched from TiDB via this status-port.
status-port = 10080
# The PD address of the cluster
pd-addr = "172.16.102.67:2379"
```

### 2. What did you expect to see? (Required)
fail and exit

### 3. What did you see instead (Required)
lightning keeps running but does nothing.

### 4. What is your TiDB version? (Required)

commit: 0d7602eb6551c9a53732be4da37d6ac2f22e375d

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.