Scatter operators can incur repeated 5-second delays during leader and joint-consensus transitions
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Description
When creating a table with pre-split Regions, scatter operators for newly
created Regions can take more than 10 seconds even when there is no meaningful
data movement, snapshot delay, disk pressure, or unhealthy peer.
In one observed case, a `scatter-region` operator took about 11 seconds:
```text
createAt: 2026-07-30 07:58:43.582 UTC
finish: 2026-07-30 07:58:54.591 UTC
takes: 11.009s
```
The operator moved peers from stores `[133, 135, 35]` to `[66, 140, 42]`:
```text
AddLearner(store 42)
AddLearner(store 66)
AddLearner(store 140)
ChangePeerV2Enter
TransferLeader(store 35 -> store 66)
ChangePeerV2Leave
RemovePeer(...)
```
The total time was dominated by two avoidable ~5-second waits in PD's operator
active-push path.
## Environment
- PD: v8.5.4
- TiKV: v8.5.4
## Observed behavior
### First delay: newly elected leader rejects the initial AddLearner proposal
The Region leader was newly elected at term 6:
```text
07:58:43.581 [raft.rs] became leader at term 6
```
PD immediately dispatched the first `AddLearner` command:
```text
07:58:43.582 [operator_controller.go] send schedule command
step="add learner peer 2583267813 on store 42"
source=create
```
TiKV rejected the `ChangePeerV2` proposal:
```text
07:58:43.582 [peer.rs] failed to propose
admin_cmd_type=Some(ChangePeerV2)
```
This is expected. In TiKV, a leader must apply an entry in its current term
before it can propose a conf change. The new leader was at term 6 while its
applied term was still term 5.
PD retried from the active-push queue only about 5.5 seconds later:
```text
07:58:49.072 [operator_controller.go] send schedule command
step="add learner peer 2583267813 on store 42"
source="active push"
07:58:49.073 [peer.rs] propose conf change peer
07:58:49.074 [apply.rs] execute admin command
```
### Second delay: TransferLeader arrives before the target applies the joint configuration
After PD observed the joint-consensus configuration, it advanced the operator
to the `TransferLeader` step:
```text
07:58:49.182 [region.go] region ConfVer changed
old-confver=3488
new-confver=3494
```
PD then immediately sent the transfer-leader command:
```text
07:58:49.183 [pd.rs] try to transfer leader
from_peer="id: 2583267698 store_id: 35 role: DemotingVoter"
to_peer="id: 2583267811 store_id: 66 role: IncomingVoter"
```
However, the target peer on store 66 rejected the request:
```text
07:58:49.183 [peer.rs] reject transferring leader
is_learner=true
wait_data=false
is_witness=false
disk_usage=Normal
pending_snapshot=false
```
The target peer locally switched to the joint configuration shortly afterward:
```text
07:58:49.185 [raft.rs] switched to configuration
raft_id=2583267811
```
This is also expected behavior. Although PD had already observed the peer as
`IncomingVoter`, the target TiKV had not yet locally applied the
joint-consensus entry and still considered itself a learner. TiKV correctly
rejects transferring leadership to a learner.
PD did not retry the transfer until about 5.4 seconds later:
```text
07:58:54.573 [pd.rs] try to transfer leader
07:58:54.573 [peer.rs] ack transfer leader
applied_index=10
committed_index=10
persist_index=10
07:58:54.574 [peer.rs] transfer leader
07:58:54.574 [raft.rs] starts to transfer leadership to 2583267811
```
The operator then quickly completed:
```text
07:58:54.591 [operator_controller.go] operator finish
takes=11.009s
```
## Root cause
TiKV's safety checks are working as intended in both cases:
1. A newly elected leader must not propose a conf change before applying the
current term.
2. A peer must not receive leadership while it is locally still a learner.
The delay comes from PD's active-push scheduling behavior.
In PD v8.5.4, the notifier interval is selected based on the current operator
step:
```go
var (
slowNotifyInterval = 5 * time.Second
fastNotifyInterval = 2 * time.Second
)
func getNextPushOperatorTime(step OpStep, now time.Time) time.Time {
nextTime := slowNotifyInterval
switch step.(type) {
case TransferLeader, PromoteLearner, ChangePeerV2Enter, ChangePeerV2Leave:
nextTime = fastNotifyInterval
}
return now.Add(nextTime)
}
```
`AddLearner` uses the 5-second slow interval. This is reasonable for actual
long-running snapshot transfers.
However, a notifier item is only assigned a new deadline when it becomes due:
```go
step := op.Check(region)
if now.Before(item.time) {
oc.opNotifierQueue.push(item)
return nil, false
}
item.time = getNextPushOperatorTime(step, now)
oc.opNotifierQueue.push(item)
return r, true
```
A Region heartbeat can advance an operator from `AddLearner` to
`TransferLeader`, but the existing notifier deadline is not brought forward
to reflect the new step.
In this case:
1. The notifier was scheduled while the operator was at `AddLearner`, using
the 5-second slow interval.
2. PD received a heartbeat that advanced the operator to `TransferLeader`.
3. The first transfer attempt was sent immediately from the heartbeat path but
was temporarily rejected because the target peer had not yet applied the
joint configuration.
4. The next active retry still waited for the earlier slow-step deadline,
rather than using the fast retry policy for `TransferLeader`.
As a result, a short-lived TiKV readiness window turns into a user-visible
multi-second delay.
## Impact
This is particularly visible for `CREATE TABLE ... PRE_SPLIT_REGIONS`:
- newly created Regions may elect leaders at roughly the same time;
- scatter operators are created in bulk;
- empty or small Regions do not need meaningful data movement;
- the fixed retry wait dominates the end-to-end DDL latency.
A single operator can accumulate multiple ~5-second waits, resulting in
pre-split table creation latency above 10 seconds.
## Expected behavior
When a Region heartbeat advances an operator to a new step, the retry deadline
should reflect the current step rather than remain tied to an earlier slow
step.
In particular, after an operator advances from `AddLearner` to
`TransferLeader`, a temporarily rejected transfer should be retried according
to the transfer-leader retry policy, without waiting for a stale `AddLearner`
deadline.
The existing TiKV safety checks should remain unchanged.
## Suggested direction
Consider improving PD's operator active-push scheduling so that a step
transition observed through a Region heartbeat can refresh or bring forward
the notifier deadline.
One possible policy is:
- when the operator step changes, schedule the next active push no later than
the retry deadline derived from the new step;
- optionally use a small bounded quick-retry budget after operator creation
and after a step transition;
- retain the existing slower retry behavior for genuinely long-running
`AddLearner` and `RemovePeer` operations after the quick-retry budget is
exhausted.
This would avoid globally reducing the retry interval for all `AddLearner`
operations while reducing unnecessary control-plane latency during short
leader-election and joint-consensus transition windows.
## Relevant code
PD v8.5.4:
- `pkg/schedule/operator/operator_controller.go`
- `slowNotifyInterval`
- `fastNotifyInterval`
- `getNextPushOperatorTime`
- `pollNeedDispatchRegion`
- `Dispatch`
- `pkg/schedule/coordinator.go`
- `pushOperatorTickInterval`
TiKV v8.5.4:
- `components/raftstore/src/store/peer.rs`
- conf change proposal requires `applied_term == current_term`
- transfer-leader target rejects learner peers
Contributor guide
Research direction
Start in pkg/schedule/operator/operator_controller.go by reading getNextPushOperatorTime, pollNeedDispatchRegion, and Dispatch, then trace how Region heartbeats update notifier deadlines. Verify the behavior around AddLearner and TransferLeader, preserving slower retries for long-running operations; done means a step transition brings the retry deadline forward without changing TiKV safety checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100