lightning failed with error “tidb lightning encountered error: fetch tso from pd failed: rpc error: code = Unknown desc = server not started” when pd rolling restart
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
1、run lightning
2、pd rolling restart
### 2. What did you expect to see? (Required)
lightning can cuccess
### 3. What did you see instead (Required)
lightning failed with error “tidb lightning encountered error: fetch tso from pd failed: rpc error: code = Unknown desc = server not started” when pd rolling restart
Run Lightning failed.
cmd start at 2024-01-16 04:03:22
cmd failed at 2024-01-16 04:10:03
stdout:
Verbose debug logs will be written to /tmp/lightning.log.2024-01-15T20.03.22Z
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| # | CHECK ITEM | TYPE | PASSED |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 1 | Source data files size is proper | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 2 | the checkpoints are valid | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 3 | table schemas are valid | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 4 | all importing tables on the target are empty | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 5 | Cluster version check passed | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 6 | Lightning has the correct storage permission | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 7 | local disk resources are rich, estimate sorted data size 26.05GiB, local available is 3.399TiB | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 8 | The storage space is rich, which TiKV/Tiflash is 5.396TiB/0B. The estimated storage space is 78.16GiB/0B. | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
|[33m 9 [0m|[33m TiKV stores (1, 13, 12, 14) contains more than 1000 empty regions respectively, which will greatly affect the import speed and suc [0m|[33m performance [0m|[33m false [0m|
|[33m [0m|[33m cess rate [0m|[33m [0m|[33m [0m|
+[33m----[0m+[33m------------------------------------------------------------------------------------------------------------------------------------[0m+[33m-------------[0m+[33m--------[0m+
| 10 | Cluster region distribution is balanced | performance | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
| 11 | no CDC or PiTR task found | critical | true |
+----+------------------------------------------------------------------------------------------------------------------------------------+-------------+--------+
tidb lightning encountered error: fetch tso from pd failed: rpc error: code = Unknown desc = server not started
[ha-test-lightning-tps-5970173-1-664.tar.gz](https://github.com/pingcap/tidb/files/13960252/ha-test-lightning-tps-5970173-1-664.tar.gz)
### 4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.0.0-alpha
Edition: Community
Git Commit Hash: https://github.com/pingcap/tidb/commit/3a3237ee49ee9f5be4c5aefb1559b97586884786
Git Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-01-15 11:42:58
GoVersion: go1.21.5
Race Enabled: false
Check Table Before Drop: false
Store: unistore
./tidb-lightning -V
Release Version: v8.0.0-alpha
Git Commit Hash: https://github.com/pingcap/tidb/commit/3a3237ee49ee9f5be4c5aefb1559b97586884786
Git Branch: heads/refs/tags/v8.0.0-alpha
Go Version: go1.21.5
UTC Build Time: 2024-01-15 11:39:25
Race Enabled: false
Contributor guide
Assessment
This issue has not been assessed yet.