Refine the load balance strategy of choosing TiFlash replica to read

Background

In TiDB, there isn't a proper way to choose follower peer or learner peer to read, which may cause hotpoint of read request.

Priviously, we used a random approach to choose TiFlash peer which can lead to balance in terms of probability.

While TiDB doesn't have ability to tell if a pure learner store(i.e. TiFlash) is down or not. Therefore if there is a TiFlash node get crashed during a query, TiDB may tend to read from the crashed node again when doing backoff which will cause a long time wait in MPP query due to some design issues. https://github.com/pingcap/tidb/pull/23589 fixed this problem in a brutal way and introduce the hotpoint issue in MPP mode.

We should find a way to solve both the load balance issue and the backoff issue.

Related work

PD is planning to design a mechanism to collect information about load(e.g. read flow, QPS, etc.) of follower peers to help schedule hotpoint introduced by stale read. They demand TiDB to choose follower peer randomly.

A keep alive mechanism is needed for doing backoff.

No files, tests, or entry points are named in the issue. Start by tracing TiFlash replica selection and backoff behavior in the related TiDB pull request, then review the planned PD load-information mechanism and keep-alive requirement; done means resolving both read-load balancing and retries after a TiFlash node failure.