syncer: improve RegionSyncer correctness and operability
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Development Task
This issue tracks the overall RegionSyncer correctness and operability improvement work.
### Background
RegionSyncer was originally designed as a lightweight region metadata sync path outside Raft. Its own comment says it ensures eventual consistency through later region heartbeats, but does not guarantee strong consistency.
That assumption was acceptable when follower region cache was mostly internal state. If a follower missed a region update, a later heartbeat for the same region could eventually repair it.
The situation changed after Active Follower, QueryRegion, and router-service related work started relying more on follower-local region cache. Follower region cache is now part of the serving path, so stale or incomplete follower state can become visible to clients.
#10666 exposed one concrete case: when a follower falls behind the leader by more than the leader-side history buffer can retain, it may miss region updates and keep serving stale region cache indefinitely.
### Problem areas
Several old assumptions need to be tightened:
- A follower may serve region data before it has finished full sync.
- A history buffer gap may silently skip updates instead of forcing a full sync.
- A failed follower stream may be removed on the leader without making the follower reconnect immediately.
- Follower-local region cache was hard to inspect or reset through HTTP APIs.
- The fixed-size history buffer is too easy to overflow on large clusters or during full sync.
- Server-side sync state is shared too broadly and does not isolate each follower well enough.
### Goals
- Make follower region cache converge safely after sync gaps.
- Prevent clients from using a follower before its region cache is ready.
- Make follower-local region cache observable and resettable.
- Make stream failures force timely reconnect and recovery.
- Reduce avoidable history-buffer overflow.
- Split follower sync state so one follower's problem does not affect the others.
### Tracking issues
#### Observed stale-cache risk
- [x] #10666
#### Recovery and readiness
- [x] #10668
- [x] #10690
#### Follower-local APIs
- [x] #10681
- [x] #10667
#### Stream lifecycle
- [x] #10684
- [x] #10731
#### History buffer
- [x] #10711
- [x] #10734
#### Per-follower sync state
- [x] #10715
### Recent related PRs
- [x] #10712
- [x] #10716
- [x] #10733
- [ ] #10735
### Completion criteria
- The follower can recover safely when its sync index is older than the available history buffer.
- Clients only use follower region reads after the follower has completed full sync.
- Operators can inspect and reset follower-local region cache when needed.
- Failed streams cause reconnect instead of leaving the follower waiting silently.
- The history buffer is sized and retained in a way that reduces repeated full-sync failures.
- The open issues above are closed or replaced by clearly linked follow-up issues.
- #10666 is closed or updated with any remaining risk after the linked work is complete.
Contributor guide
Research direction
Start by reviewing the RegionSyncer background, completion criteria, tracking issues #10666, #10668, #10690, #10681, #10667, #10684, #10731, #10711, #10734, and #10715, plus the related PRs. Determine which follow-up work remains open and how the changes should be coordinated. Done means the listed recovery, readiness, observability, reconnect, history-buffer, and per-follower-state goals are addressed or replaced by clearly linked issues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100