tikv / tikv/pd

syncer: make region history buffer adaptively resize

Open
#10,711 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Enhancement Task

The region syncer history buffer can still overflow during a full region sync when the PD leader continues receiving region heartbeats and the history index advances faster than the follower can finish the full snapshot and catch-up phase.

#10692 improves the initial history buffer capacity by scaling it with visible memory, but the buffer capacity is still fixed after startup. A fixed capacity can reduce the overflow probability, but it cannot prevent repeated full-sync retries when the actual required history window becomes larger than the current buffer.

This issue tracks an adaptive history buffer implementation for the region syncer.

### Goal

Make the region syncer history buffer resize according to the actual required history window.

Expected behavior:

- Keep the default/base history buffer capacity memory-aware, following the direction in #10692.
- Keep the runtime capacity in units of `2^n * 10000`.
- Grow the buffer when the required replay window stays above the healthy range.
- Shrink the buffer when the required replay window stays below the healthy range for a while.
- Keep the healthy usage range around `(capacity / 4, capacity / 2]`.
- Use 2x growth and 1/2 shrink steps.
- Avoid triggering growth merely because the ring buffer is full; use the required replay window instead.
- During full region sync, retain the history range needed for catch-up so the leader does not overwrite the change logs required by the follower before the full sync completes.
- Keep a memory-based maximum capacity to avoid unbounded memory usage.
- If the required retained range exceeds the maximum capacity, fail the current full sync attempt explicitly and keep follower local reads disabled.

### Suggested implementation direction

- Add adaptive resize support to `historyBuffer`.
- Track the required history window from:
- full sync retain start index,
- follower requested `StartIndex`,
- replay window observed during normal sync,
- history miss / full-sync fallback cases.
- Resize capacity to `2^n * 10000`.
- Use the memory-based sizing from #10692 as the base capacity source.
- Add a memory-based maximum capacity higher than the base capacity.
- Only shrink when there is no active full-sync retain.
- Keep resize logic encapsulated inside `historyBuffer` as much as possible.

### Validation

Add tests for:

- Normal ring behavior when there is no retain.
- Growth when the required replay window exceeds the healthy range.
- Shrink when the required replay window remains below the healthy range.
- Capacity is always `2^n * 10000`.
- Full sync retains the required catch-up history.
- Multiple concurrent retain windows preserve the earliest required index.
- Full sync succeeds when history grows beyond the base capacity but stays below the maximum capacity.
- Full sync fails explicitly without enabling follower local reads when the required retained history exceeds the maximum capacity.
- Existing incremental sync behavior remains unchanged when the follower is within the available history window.

### PR split

First PR:

- Adaptive history buffer core logic.
- Full-sync retain behavior.
- Unit and syncer tests.

Second PR:

- Metrics and dashboard updates.

Contributor guide

Open the contributing guide

Research direction

Start at the historyBuffer implementation and the full-sync retain path, then inspect the existing unit and syncer tests. Done means adaptive capacities use the specified power-of-two units and memory limits, retain windows are preserved, normal sync remains unchanged, and over-limit full sync fails without enabling follower local reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.