syncer: make region history buffer adaptively resize
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Enhancement Task
The region syncer history buffer can still overflow during a full region sync when the PD leader continues receiving region heartbeats and the history index advances faster than the follower can finish the full snapshot and catch-up phase.
#10692 improves the initial history buffer capacity by scaling it with visible memory, but the buffer capacity is still fixed after startup. A fixed capacity can reduce the overflow probability, but it cannot prevent repeated full-sync retries when the actual required history window becomes larger than the current buffer.
This issue tracks an adaptive history buffer implementation for the region syncer.
### Goal
Make the region syncer history buffer resize according to the actual required history window.
Expected behavior:
- Keep the default/base history buffer capacity memory-aware, following the direction in #10692.
- Keep the runtime capacity in units of `2^n * 10000`.
- Grow the buffer when the required replay window stays above the healthy range.
- Shrink the buffer when the required replay window stays below the healthy range for a while.
- Keep the healthy usage range around `(capacity / 4, capacity / 2]`.
- Use 2x growth and 1/2 shrink steps.
- Avoid triggering growth merely because the ring buffer is full; use the required replay window instead.
- During full region sync, retain the history range needed for catch-up so the leader does not overwrite the change logs required by the follower before the full sync completes.
- Keep a memory-based maximum capacity to avoid unbounded memory usage.
- If the required retained range exceeds the maximum capacity, fail the current full sync attempt explicitly and keep follower local reads disabled.
### Suggested implementation direction
- Add adaptive resize support to `historyBuffer`.
- Track the required history window from:
- full sync retain start index,
- follower requested `StartIndex`,
- replay window observed during normal sync,
- history miss / full-sync fallback cases.
- Resize capacity to `2^n * 10000`.
- Use the memory-based sizing from #10692 as the base capacity source.
- Add a memory-based maximum capacity higher than the base capacity.
- Only shrink when there is no active full-sync retain.
- Keep resize logic encapsulated inside `historyBuffer` as much as possible.
### Validation
Add tests for:
- Normal ring behavior when there is no retain.
- Growth when the required replay window exceeds the healthy range.
- Shrink when the required replay window remains below the healthy range.
- Capacity is always `2^n * 10000`.
- Full sync retains the required catch-up history.
- Multiple concurrent retain windows preserve the earliest required index.
- Full sync succeeds when history grows beyond the base capacity but stays below the maximum capacity.
- Full sync fails explicitly without enabling follower local reads when the required retained history exceeds the maximum capacity.
- Existing incremental sync behavior remains unchanged when the follower is within the available history window.
### PR split
First PR:
- Adaptive history buffer core logic.
- Full-sync retain behavior.
- Unit and syncer tests.
Second PR:
- Metrics and dashboard updates.
Contributor guide
Research direction
Start at the historyBuffer implementation and the full-sync retain path, then inspect the existing unit and syncer tests. Done means adaptive capacities use the specified power-of-two units and memory limits, retain windows are preserved, normal sync remains unchanged, and over-limit full sync fails without enabling follower local reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100