Lightning stuck after tikv is down
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1. dump table using dumpling
2. restore table with lightning: `tidb-lightning --config ./lightning.toml`
3. tikv crashed for some reasons
4. lightning got stuck
lightning log:
```
Verbose debug logs will be written to /tmp/tidb-lightning.log
+---+----------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+---+----------------------------------------------+-------------+--------+
| 1 | Source csv files size is proper | performance | true |
+---+----------------------------------------------+-------------+--------+
| 2 | checkpoints are valid | critical | true |
+---+----------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+---+----------------------------------------------+-------------+--------+
| 4 | Cluster is available | critical | true |
+---+----------------------------------------------+-------------+--------+
| 5 | Lightning has the correct storage permission | critical | true |
+---+----------------------------------------------+-------------+--------+
[2022/04/11 08:10:02.441 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40052->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.441 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40050->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40048->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40046->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40038->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40042->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40036->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40040->10.244.5.105:20160: write: connection reset by peer"] [system=grpc] [grpc_log=true]
[2022/04/11 08:10:02.442 +00:00] [ERROR] [grpclogger.go:116] ["[transport]transport: loopyWriter.run returning. Err: write tcp 172.17.0.2:40030->10.244.5.105:20160: write: broken pipe"] [system=grpc] [grpc_log=true]
```
lightning.toml
```
[lightning]
# Logging
file = "/tmp/tidb-lightning.log"
level = "info"
[tikv-importer]
# Uses the Local-backend
backend = "local"
# Sets the directory for temporarily storing the sorted key-value pairs.
# The target directory must be empty.
sorted-kv-dir = "/tmp/sst"
[mydumper]
# Local source data directory
data-source-dir = "/tmp/dumpling"
# Configures the wildcard rule. By default, all tables in the mysql, sys, INFORMATION_SCHEMA, PERFORMANCE_SCHEMA, METRICS_SCHEMA, and INSPECTION_SCHEMA system databases are filtered.
# If this item is not configured, the "cannot find schema" error occurs when system tables are imported.
filter = ['*.*', '!mysql.*', '!sys.*', '!INFORMATION_SCHEMA.*', '!PERFORMANCE_SCHEMA.*', '!METRICS_SCHEMA.*', '!INSPECTION_SCHEMA.*']
[tidb]
# Information of the target cluster
host = "172.16.102.67"
port = 4000
user = "root"
# Table schema information is fetched from TiDB via this status-port.
status-port = 10080
# The PD address of the cluster
pd-addr = "172.16.102.67:2379"
```
### 2. What did you expect to see? (Required)
fail and exit
### 3. What did you see instead (Required)
lightning keeps running but does nothing.
### 4. What is your TiDB version? (Required)
commit: 0d7602eb6551c9a53732be4da37d6ac2f22e375d
Contributor guide
Assessment
This issue has not been assessed yet.