pingcap / pingcap/tidb

more flexible follower read dispatch strategy

Open
#35,926 3 comments 0 reactions 1 assignee Claimed by @glorv View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

In the cloud environment, the tidb clusters are commonly deployed across multiple "Access Zone"(AZ for short)s. Because the traffic cost across different AZ is much higher than within the same AZ. We can reduce the cost by dispatching read request to
the TiKV/TiFlash with a higher priority.

#28033 introduced a new `tidb_replica_read` option "closest-replica" that always dispatch all read request to the closest TiKV. But since the "follower read" feature need to propose an extra "readIndex" message through the raft state machine. This overhead may cause performance regression. For cop requests with small response, read from local AZ can not save the traffic because the extra "readIndex" messages are always cross AZ traffic. And the extra cpu usage to process these raft message can hurt the performance.

I propose to add a new option "closest-adaptive" for `tidb_replica_read` variable. This new option tries to dispatch request to the nearest store only when the response size is bigger than a certain threshold which is define by a new variable "tidb_adaptive_closest_read_threshold".

BTW, when the cop traffic from tidbs in different AZ are not evenly distributed, it may cause high load in some store while other are low loaded. This can cause high tail-lantency and low overall throughput. So when follow-read is enabled, the tikv client may need to track the load of each storage and force fallback to leader read when some stores' load are high.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.