Follower read target selection should consult PD for store recovery, not only TiKV's self-reported slow score
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 362
- Forks
- 274
- Avg merge
- 4d 19m
- Merged PRs (30d)
- 12
Description
Background
When a TiKV store hits a transient problem (e.g. a faulty/jittery disk), PD marks it as a slow store and evicts its leaders. Follower stale-read traffic, however, can still be routed to that store while it is recovering.
Current Behavior
When client-go selects the target replica for a follower read, it relies on the slow score that the TiKV store reports about itself to decide whether the store has recovered and is eligible to serve reads again.
The problem is that a recovered self-reported slow score does not guarantee the store is actually ready to take traffic:
- The signal is local to the store and can be noisy/flapping
- It is not coordinated with PD's view of the store's recovery / leader-eviction state.
As a result, follower reads can be sent back to a not-yet-recovered store too early, hotspotting it.
Expected Behavior
Before routing a follower (stale) read to a store that was recently slow/recovering, client-go should consult PD to confirm the store has fully recovered (e.g. that PD no longer considers it a slow store / has stopped evicting its leaders), rather than relying solely on the store's self-reported slow score.
Proposed Direction
- Treat PD's store recovery state as the authoritative signal for "is this store ready to serve reads again," and gate follower-read target selection on it.
- Keep the TiKV self-reported slow score as a fast local hint, but require agreement with PD's view before sending traffic back to a recently-slow store.
- Consider a hysteresis / cool-down so a flapping slow score cannot repeatedly re-add a store as a follower-read target.
Related
- Related PD behavior: tikv/pd#9384 — Recovery time is not expected in slow store scheduler (https://github.com/tikv/pd/issues/9384) (leaders transferred back prematurely when recovery time < jitter duration).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing follower-read target selection in client-go and how it consumes TiKV's self-reported slow score. Then inspect the PD store recovery and leader-eviction state described in the issue. Done means recently slow stores are not selected again until PD confirms recovery, with the local score retained as a hint and flapping addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100