tikv / tikv/pd

PD may not be able to sense the change of internal raft leader in etcd in time

Open
#7,780 2 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

```shell
pd-2.log:[2024/01/28 11:13:33.947 +08:00] [INFO] [raft.go:706] ["fef5444e2c4d3d9c became follower at term 4"]
pd-1.log:[2024/01/28 11:13:33.953 +08:00] [INFO] [raft.go:771] ["a22604cca51ee334 became leader at term 4"]
```

The etcd leader of pd-2 dropped at 11:13:33.947, and then the etcd leader was elected by pd-1 as the new etcd leader. However, it was not until 11:18:30.646 that the PD leader of pd-2 stepped down, and pd-1 was elected as the new PD leader.

```shell
pd-2.log:[2024/01/28 11:18:30.646 +08:00] [INFO] [server.go:1687] ["etcd leader changed, resigns pd leadership"] [old-pd-leader-name=tc-pd-2]
pd-1.log:[2024/01/28 11:18:31.627 +08:00] [INFO] [server.go:1529] ["pd leader has changed, try to re-campaign a pd leader"]
```

The only check for the PD leader to find out the etcd leader is changed:

https://github.com/tikv/pd/blob/1c5486521e4794d6e02868763388937c40943887/server/server.go#L1805-L1809

So it's reasonable to conclude that `m.etcd.Server.Lead()` may not return the latest etcd leader as soon as possible.

In the etcd code, there is a small detail inside the raft `Ready` preparation:

https://github.com/etcd-io/etcd/blob/85b640cee793e25f3837c47200089d14a8392dc7/raft/node.go#L311-L322

So it is possible that the raw raft node has already finished the election but the upper etcd server can not get the latest soft state to apply, which causes the new etcd leader to be set in a long time later. This could happen more easily, especially when we have some chaos like IO hang injected into the etcd.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.