Address the compatibility issue with hibernate region

Known issues (2022 Q1)

• https://github.com/pingcap/tiflow/issues/4461

Feature Request

Is your feature request related to a problem? Please describe:

Background
Hibernate Region is a feature for Raft. It stops to send heartbeats after a Raft group have not received proposals for a while. (see more details in 2019-03-04-hibernate-raft.md)
And in TiCDC, it needs a timestamp (called resolvedTS) in which we can ensure all transactions which start before this timestamp have completed and been sent from TiKV to TiCDC. TiKV advances by tracking the locks which haven't been clean, and getting a tso from pd periodically (1s by default), and the maximum timestamp among the start ts for the locks and the tso is region resolvedTS. TiCDC uses the maximum timestamp among region resolvedTSs as global resolvedTS.

Problem
As was shown above, it's easy to know the region resolvedTS is ought to be advanced by the leader of a Raft group. We must check the leadership before advancing the region resolvedTS, so that Hibernate Region was broken since we cannot know whether a peer is a leader without sending a heartbeat after the lease was expired.

TODO list

• In Raft, a peer is a leader means this peer was accepted by the majority of the peers in the Raft group.
  The key is majority, we can check the majority (or leadership) at the client-side (in this context client means TiCDC). In this way, we avoid to send a heartbeat inside the Raft group, then it will not break Hibernate Region.

Teachability, Documentation, Adoption, Migration Strategy:

TODO

Value

Value description

Very important in large scale cluster

Value score

• (TBD) / 5

Workload estimation

• (TBD) 60 person-day

Time

GanttStart: 2020-09-01
GanttDue: 2020-10-30
GanttProgress: 60%