tikv / tikv/pd

merge operator is not canceled when the region becomes unhealthy, blocking down-peer replacement until operator timeout (~28m)

Open
#11,142 3 comments 0 reactions 1 assignee Claimed by @bufferflies View on GitHub
type/enhancement
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?

A 12-TiKV dr-auto-sync cluster (primary dc1 + dr dc2, 5 replicas, replication-mode=dr-auto-sync) was running.

Operations performed on the cluster:

1. Put the cluster into sync_recover state: isolated 2 DR nodes (tikv4, tikv5) via network loss to degrade the cluster to async mode, waited 30 minutes, then recovered their network so the cluster entered sync_recover.
2. Downed a primary node (tikv1) via network isolation, and waited for region health to recover.

(Reproduction sequence with internal events, timezone +08:00:)

- 22:28:25 PD created merge operator `merge-region {merge: region 354 to 361}`. At this moment region 354 was healthy (no down/pending peers).
- 22:28:26 region 354 got a down peer (peer 8784 on store 6, network isolated).
- 22:28:26 ~ 22:51:26 PD kept sending the merge step (`merge region 354 into region 361`) to region 354 (236 active-push commands), 0 finish.
- During the whole period region 354 had no `replace-rule-down-peer` operator at all.
- down-peer-region-count stayed at 2 for ~11 minutes (22:39:30 ~ 22:50:30), until the merge operator hit its timeout (28m) and was removed; only then the down peer was replaced.

### What did you expect to see?

When a region becomes unhealthy (has a down/pending peer) after a merge operator was created but before it finishes, PD should cancel the stuck merge operator so that the replica checker can replace the down peer, and retry the merge naturally after the region recovers.

### What did you see instead?

The merge operator stayed pending for ~28 minutes (timeout:28m0s). During that window the down peer could not be replaced and the replica gap persisted; in the DR scenario the cluster stayed in sync_recover and could not return to sync, so DR protection was degraded.

Evidence from pd.log:

- 22:28:25 `add operator` `merge-region {merge: region 354 to 361}` `timeout:[28m0s]`
- 22:28:26 `region has down peer on connected store` `[down-peer=8784]` `[store-id=6]`
- 22:28:26~22:51:26 repeated `send schedule command` `[step="merge region 354 into region 361"]` (source=active push), no operator finish for region 354
- 0 `replace-rule-down-peer` operators created for region 354 during the whole window
- `pd_regions_status{type="down-peer-region-count"}` stayed at 2 for ~11 minutes until the merge operator timed out

### What version of PD are you using (`pd-server -V`)?

```text
Release Version: v9.0.0-beta.2.pre-473-gbe1e87f
Git Hash: be1e87f238f2b88aad9d55057cd12b82e6c66f92
Build Time: 2026-08-11 03:06:09
```

### Root cause analysis (suspected)

The merge operator is only created on healthy regions, but once created it is not canceled when the region becomes unhealthy:

- Merge is created only on healthy regions: `merge_checker.go:123` `if !filter.IsRegionHealthy(region) { return nil }` (IsRegionHealthy = no down/pending peers). So the merge operator can only be created before the down peer appears.
- Down-peer replacement is short-circuited: `checker_controller.go:401` `tryAddOperators` returns immediately when the region already has any operator, so rule_checker's down-peer replacement can never be generated.
- Merge cannot finish and PD does not cancel it proactively: on the active side `MergeRegion.IsFinish` always returns false (it waits for the region to disappear after TiKV executes the merge), while the down peer prevents the merge from progressing, so the operator can only be removed by its fixed timeout (`operator.go:329 CheckTimeout`).
- Repeated create-reject loop: rule_checker keeps generating replacement operators each round, rejected as `already exist` by `operator_controller.go:466`; 61394 cancel events observed.

Suggested fix (option 1, preferred): when a region that already has a merge operator becomes unhealthy (GetDownPeers() non-empty), cancel the pending merge operator (including the related/passive side via the existing RelatedMergeRegion cancel mechanism), so rule_checker can replace the down peer, and merge_checker will naturally recreate the merge after the region recovers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.