tikv / tikv/pd

Lighting failed when kill pd leader

Open
#9,142 2 comments 0 reactions 1 assignee Claimed by @okJiang View on GitHub
may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/major type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
1、Lighting import
2、kill pd leader

### What did you expect to see?
Lighting can succeed

### What did you see instead?
Lighting failed
`[2025/03/09 23:34:46.447 +00:00] [ERROR] [service_discovery.go:581] ["[pd] failed to update service mode"] [urls="[http://tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379,http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379,http://tc-pd-2.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379]"] [error="[PD:client:ErrClientGetClusterInfo]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE"] [stack="github.com/tikv/pd/client/servicediscovery.(*serviceDiscovery).updateServiceModeLoop\n\t/root/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20250213082949-e8930327be42/servicediscovery/service_discovery.go:581"]
[2025/03/09 23:34:46.447 +00:00] [INFO] [service_discovery.go:889] ["[pd] cannot update member from this url"] [url=http://tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE"]
[2025/03/09 23:34:46.448 +00:00] [INFO] [client.go:210] ["[tso] switch the tso leader serving url"] [new-url=http://tc-pd-2.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379]
[2025/03/09 23:34:46.448 +00:00] [INFO] [service_discovery.go:986] ["[pd] switch leader"] [new-leader=http://tc-pd-2.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379] [old-leader=http://tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379]
[2025/03/09 23:34:46.448 +00:00] [INFO] [service_discovery.go:889] ["[pd] cannot update member from this url"] [url=http://tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.233.76.189:2379: connect: connection refused\" target:tc-pd-0.tc-pd-peer.ha-test-lightning-tps-7783261-1-181.svc:2379 status:TRANSIENT_FAILURE"]
[2025/03/09 23:36:37.833 +00:00] [INFO] [import.go:1134] [progress] [total=65.7%] [tables="11/12 (91.7%)"] [chunks="319/578 (55.2%)"] [engines="22/24 (91.7%)"] [restore-bytes=15.92GiB/39.97GiB] [restore-rows=257940402/647796346(estimated)] [import-bytes=63.63GiB/159.8GiB(estimated)] ["encode speed(MiB/s)"=18.109545168332534] [state=writing] [remaining=7m49s]
[2025/03/09 23:37:38.992 +00:00] [INFO] [table_import.go:1410] ["analyze completed"] [table=`location`.`MonsterSource`] [takeTime=4m45.311751634s] []
[2025/03/09 23:37:38.993 +00:00] [ERROR] [import.go:1413] ["restore all tables data failed"] [takeTime=16m1.172135185s] [error="[Lightning:Restore:ErrRestoreTable]restore table `location`.`GooglePoiSource` failed: rpc error: code = Unavailable desc = error reading from server: EOF"]
[2025/03/09 23:37:38.993 +00:00] [INFO] [import.go:1008] ["everything imported, stopping periodic actions"]
[2025/03/09 23:37:38.995 +00:00] [ERROR] [import.go:577] ["run failed"] [step=4] [error="[Lightning:Restore:ErrRestoreTable]restore table `location`.`GooglePoiSource` failed: rpc error: code = Unavailable desc = error reading from server: EOF"]
[2025/03/09 23:37:38.995 +00:00] [ERROR] [import.go:587] ["the whole procedure failed"] [takeTime=16m4.489249975s] [error="[Lightning:Restore:ErrRestoreTable]restore table `location`.`GooglePoiSource` failed: rpc error: code = Unavailable desc = error reading from server: EOF"]
[2025/03/09 23:37:39.027 +00:00] [INFO] [service_discovery.go:544] ["[pd] exit member loop due to context canceled"]`

### What version of PD are you using (`pd-server -V`)?
./pd-server -V
Release Version: v9.0.0-alpha-70-g5e82f16
Edition: Community
Git Commit Hash: 5e82f16ecf4e4ecfce8f02fce8c5022a26c7f104
Git Branch: HEAD
UTC Build Time: 2025-03-06 10:18:40
2025-03-10T07:21:32.242+0800

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.