tikv / tikv/pd

balance-region-scheduler can oscillate when fast operator influence is overwritten per region

Open
#11,133 3 comments 0 reactions 0 assignees View on GitHub
component/scheduler contribution severity/moderate type/bug
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

Investigated repeated `balance-region-scheduler` callbacks where completed move-peer operators for the same region were reversed within the fast-operator influence window.

A minimal deterministic reproduction should be possible with the following setup:

1. Create several stores with a small number of regions so that moving one large region changes the source and target scores significantly.
2. Complete a fast move-peer operator for region `R`, for example `S1 -> S2`.
3. Before `FastOperatorFinishTime` expires, complete another fast move-peer operator for a different replica of the same region, for example `S3 -> S4`.
4. Run `balance-region-scheduler` again and inspect the fast-finished operator influence for `R`.

The observed scheduling pattern was equivalent to:

```text
S1 -> S2
S2 -> S1 # about four seconds later
S1 -> S2 # about five seconds later
```

Each operator completed successfully; this was not an operator timeout or a retry of one failed operator.

Related context: #3162 discusses redundant operators, and #3727 introduced fast-finished operator influence specifically to avoid scheduling callbacks such as `A -> B, B -> C, C -> A`. This report is narrower: multiple fast operators for one region overwrite each other before influence is calculated.

### What did you expect to see?

PD should retain the net influence of all relevant fast-finished move-peer operators for the same region during the influence TTL. The scheduler should not immediately reverse a completed move solely because a later operator for another replica overwrote the earlier movement history.

### What did you see instead?

`fastOperators` is a TTL map keyed only by region ID:

- [`pushFastOperator`](https://github.com/tikv/pd/blob/b7828f1f0871b6adefcaba102cc36f2793ec0f73/pkg/schedule/operator/operator_controller.go#L840-L842) calls `fastOperators.Put(op.RegionID(), op)`.
- [`ttlCache.putWithTTL`](https://github.com/tikv/pd/blob/b7828f1f0871b6adefcaba102cc36f2793ec0f73/pkg/cache/ttl.go#L65-L74) replaces the existing value for that key.
- [`GetFastOpInfluence`](https://github.com/tikv/pd/blob/b7828f1f0871b6adefcaba102cc36f2793ec0f73/pkg/schedule/operator/operator_controller.go#L928-L940) therefore sees at most one fast-finished operator per region.

When several replicas of one region move in quick succession, each new operator replaces the previous operator in the cache and resets its TTL. Earlier source/target influence is lost. Store scores can then flip after a large-region move, allowing `balance-region-scheduler` to create an inverse operator within seconds. Repetition can produce sustained operator churn, unnecessary snapshot/peer work, and transient stale-region retries in clients.

### Causal chain

1. A region merge changes region size and store distribution enough to make several stores strong balance candidates.
2. `balance-region-scheduler` begins moving replicas rapidly. The individual operators complete normally and often finish in tens of milliseconds.
3. Because each operator finishes in less than `FastOperatorFinishTime`, PD records it as a fast-finished operator.
4. Several different replicas of the same region move before the influence TTL expires. Every new operator uses the same region ID as the cache key and overwrites the preceding operator.
5. `GetFastOpInfluence` therefore omits part of the recent net movement. After a large region changes stores, the raw source and target scores exchange order, and PD can accept the inverse move only seconds later.
6. Reversing peer placement removes a peer and may shortly recreate the replica on the same store with a new peer ID. Clients holding the previous region metadata can race this transition and briefly address the removed peer.
7. In the observed client/server stack, that stale-peer response was handled through a generic compatibility path and retried up to ten times, amplifying one PD placement callback into repeated requests and INFO logs. This last amplification is outside PD, but the repeated peer turnover originates from the scheduling oscillation.

### Sanitized timeline

All timestamps below are UTC. Region, store, peer, host, and deployment identifiers are intentionally omitted or replaced with synthetic names.

- `2026-08-06 08:50:00` to `08:59:46`: no `balance-region-scheduler` operator additions were observed.
- `08:59:42.826`: merge checking started for a small region merging into a much larger target region.
- `08:59:45.883`: the merge completed.
- `08:59:46.148`: only 265 ms later, PD created the first balance-region operator. Four operators were created within the next 33 ms, including one for the newly merged region.
- By `09:02:15`: new balance-region operator rate had risen to approximately 3.3 operators per second.
- `09:24:55` to `09:24:59`: a large region moved several replicas through multiple stores and then reversed the net placement in less than 4.2 seconds.
- `09:25:09`: another region moved `S1 -> S2`; about four seconds later PD scheduled `S2 -> S1`, and about five seconds after that it scheduled `S1 -> S2` again.
- `09:25:23`: the first downstream stale-peer mismatch became visible shortly after the peer replacement.
- `2026-08-11 06:56:07`: in a later recurrence, the old peer was destroyed, a replacement peer was registered on the same store about 292 ms later, and a client reported the stale peer ID about 102 ms after replacement registration.
- The condition recurred intermittently over several days and generated thousands of downstream INFO records, while the individual PD operators continued to finish successfully.

### Why it started

The strongest initiating signal is the region merge immediately before the quiet-to-burst transition. The newly merged target participated in the first group of balance moves, and no balance-region operators were present before the merge completed. This makes the merge the most likely perturbation, but not a proven root trigger because historical scheduler debug metrics and a dry-run decision trace were unavailable.

The sustained oscillation is better supported: a large-region move changes the raw store scores sharply, while the one-entry-per-region fast-operator cache forgets earlier moves when another replica of the same region moves. The scheduler then sees the source and target scores reverse without the full recent movement influence that #3727 intended to preserve. A successful scheduling decision resets the scheduler interval to its fast path, allowing the incomplete influence to produce a burst of additional operators.

The onset did not correlate with a scheduler configuration change, scheduler pause/resume, placement-rule update, store state transition, process restart, resource resize, unavailable region, low-space condition, or CPU/memory/I/O saturation. This makes a code-level scheduling limitation more likely than an operational configuration or infrastructure event.

Confidence is high for the cache-overwrite mechanism that sustains the oscillation, and medium for the preceding merge as the initiating perturbation.

### What version of PD are you using (`pd-server -V`)?

Observed on PD commit [`b7828f1f0871b6adefcaba102cc36f2793ec0f73`](https://github.com/tikv/pd/commit/b7828f1f0871b6adefcaba102cc36f2793ec0f73). The relevant one-entry-per-region cache behavior is also present in current code.

### Suggested fix and tests

Consider retaining cumulative recent movement influence per region, or otherwise preserving every relevant fast-finished move during the TTL, instead of storing only the latest operator for a region.

A regression test could push two or more completed move-peer operators for the same region, verify that `GetFastOpInfluence` includes the net effect of every move, and assert that `balance-region-scheduler` does not generate the immediate inverse placement while that influence is active.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/schedule/operator/operator_controller.go at pushFastOperator and GetFastOpInfluence, then inspect pkg/cache/ttl.go to understand the replacement behavior. Add a regression test for multiple completed move-peer operators on one region, and verify the influence includes their net movement and prevents an immediate inverse balance decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.