PD leader is abnormal when rolling restart
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
1. Deploy tidb cluster (with ticdc node) using operator.
2. Rolling restart pd [via configuration update](https://docs.pingcap.com/tidb-in-kubernetes/dev/restart-a-tidb-cluster#:~:text=Delete%0A%20%20pd%3A%0A%20%20%20%20...-,annotations,-%3A%0A%20%20%20%20%20%20tidb.pingcap).
### What did you expect to see?
the cluster is normal
### What did you see instead?
```log
[2024/03/17 03:44:29.073 +00:00] [INFO] [server.go:1813] [\"no longer a leader because lease has expired, pd leader will step down\"]
```
1. The session between PD and TiCDC is disconnected
```log
[2024/03/17 03:44:31.716 +00:00] [WARN] [capture.go:604] ["session is disconnected"] [error="[CDC:ErrEtcdSessionDone]the etcd session is done"] [errorVerbose="[CDC:ErrEtcdSessionDone]the etcd session is done\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run\n\tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:190\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:595\ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).campaignOwner.func4\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:531\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.6.0/errgroup/errgroup.go:78\nruntime.goexit\n\truntime/asm_amd64.s:1650](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/errors@v0.11.5-0.20231212100244-799fae176cfb/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/orchestrator.(*EtcdWorker).Run/n/tgithub.com/pingcap/tiflow/pkg/orchestrator/etcd_worker.go:190/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).runEtcdWorker/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:595/ngithub.com/pingcap/tiflow/cdc/capture.(*captureImpl).campaignOwner.func4/n/tgithub.com/pingcap/tiflow/cdc/capture/capture.go:531/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/tgolang.org/x/sync@v0.6.0/errgroup/errgroup.go:78/nruntime.goexit/n/truntime/asm_amd64.s:1650)"]
```
2. The session between PD and TiDB is disconnected
```log
[2024/03/17 03:44:33.859 +00:00] [INFO] [manager.go:267] [\"etcd session is done, creates a new one\"] [\"owner info\"=\"[autoid] tidb/autoid/leader ownerManager upstream-tidb-1.upstream-tidb-peer.cdc-airbnb-simple-tian-gcp-tps-7500530-1-808.svc:10080\"]"}
```
### What version of PD are you using (`pd-server -V`)?
6ee25f9f8177067957b0932f1be7e2cace0202d0
Contributor guide
Assessment
This issue has not been assessed yet.