Improve the success rate of operator execution when scatting or other important scenes

## Feature Request

### Describe your feature request related problem
Since TiKV is partitioning by key range, the hotspot problem has always been a headache. we have done many things to solve it such as to improve hot region scheduler, add `shards_bits` in table meta, supports `auto_random`, and so on. But in some special scenes, especially the scene of the statistical settlement of the banking business. we need to know the range of hotspots in advance and pre-partition them and then evenly distribute the partitions to all nodes，this is what [`pre_split` and `split table`](https://docs.pingcap.com/zh/tidb/stable/sql-statement-split-region#split-region-%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3) do.

user will use `split table` with specifying boundaries, such as:
```
SPLIT TABLE t INDEX idx1 BETWEEN ("a") AND ("z") REGIONS 25;
```
behind this statement, TiDB actively calls the TiKV interface to split the region. and then call PD to scatter those regions. However, because the scatter is a scheduling behavior, and it is asynchronous, it will often happen that the execution fails during the do the operator process. which will cause performance stability problems. such as the region re-election，the operator may failed:
```
"scatter-region {mv light peer: store [14 5054383 13] to [5054381 87185004 19]} (kind:leader,region, region:89471081(2223167,2108), createAt:2020-07-17 09:44:46.315172159 +0800 CST m=+40923.043939740, startAt:2020-07-17 09:44:46.31528439 +0800 CST m=+40923.044051971, currentStep:10, steps:[add learner peer 89476136 on store 19, promote learner peer 89476136 on store 19 to voter, transfer leader from store 14 to store 13, remove peer on store 14, add learner peer 89476134 on store 5054381, promote learner peer 89476134 on store 5054381 to voter, remove peer on store 5054383, add learner peer 89476135 on store 87185004, promote learner peer 89476135 on store 87185004 to voter, transfer leader from store 13 to store 19, remove peer on store 13])\""] [reason="cannot remove leader peer"]
pd_10_131_10_244_2379.log:130451:[2020/07/17 09:44:48.937 +08:00] [Info] [operator_controller.go:567] ["operator canceled"] [region-id=89470585] [takes=3.767084723s] [operator="\"scatter-region {mv light peer: store [5054383 13 14] to [12 5054377 22]} (kind:leader,region, region:89470585(2223167,2108), createAt:2020-07-17 09:44:45.170149551 +0800 CST m=+40921.898917134, startAt:2020-07-17 09:44:45.170473864 +0800 CST m=+40921.899241451, currentStep:9, steps:[add learner peer 89475781 on store 12, promote learner peer 89475781 on store 12 to voter, remove peer on store 13, add learner peer 89475780 on store 22, promote learner peer 89475780 on store 22 to voter, remove peer on store 5054383, add learner peer 89475779 on store 5054377, promote learner peer 89475779 on store 5054377 to voter, transfer leader from store 14 to store 12, remove peer on store 14])\""] [reason="cannot remove leader peer"]
pd_10_131_10_244_2379.log:139199:[2020/07/17 09:44:50.683 +08:00] [Info] [operator_controller.go:567] ["operator canceled"] [region-id=89472041] [takes=2.43430889s] [operator="\"scatter-region {mv light peer: store [14 5054383 13] to [17 87185011 87185002]} (kind:leader,region, region:89472041(2223167,2108), createAt:2020-07-17 09:44:48.248532068 +0800 CST m=+40924.977299676, startAt:2020-07-17 09:44:48.24907874 +0800 CST m=+40924.977846330, currentStep:10, steps:[add learner peer 89476815 on store 17, promote learner peer 89476815 on store 17 to voter, transfer leader from store 14 to store 13, remove peer on store 14, add learner peer 89476816 on store 87185011, promote learner peer 89476816 on store 87185011 to voter, remove peer on store 5054383, add learner peer 89476814 on store 87185002, promote learner peer 89476814 on store 87185002 to voter, transfer leader from store 13 to store 17, remove peer on store 13])\""] [reason="cannot remove leader peer"]

❯ag "operator canceled" |grep "reason" | grep "scatter"| wc -l
95
```
and the reason show as the leader changed to the peer that we want to removed.
```
pd_10_131_10_244_2379.log:[2020/07/17 09:44:48.248 +08:00] [Info] [operator_controller.go:424] ["add operator"] [region-id=89472041] [operator="\"scatter-region {mv light peer: store [14 5054383 13] to [17 87185011 87185002]} (kind:leader,region, region:89472041(2223167,2108), createAt:2020-07-17 09:44:48.248532068 +0800 CST m=+40924.977299676, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, steps:[add learner peer 89476815 on store 17, promote learner peer 89476815 on store 17 to voter, transfer leader from store 14 to store 13, remove peer on store 14, add learner peer 89476816 on store 87185011, promote learner peer 89476816 on store 87185011 to voter, remove peer on store 5054383, add learner peer 89476814 on store 87185002, promote learner peer 89476814 on store 87185002 to voter, transfer leader from store 13 to store 17, remove peer on store 13])\""]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:48.249 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476815 on store 17"] [source=create]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:48.318 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Add peer:{id:89476815 store_id:17 is_learner:true }"] [old-confver=2108] [new-confver=2109]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:48.318 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476815 on store 17"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.258 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="promote learner peer 89476815 on store 17 to voter"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.274 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89476815 store_id:17 is_learner:true },Add peer:{id:89476815 store_id:17 }"] [old-confver=2109] [new-confver=2110]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.275 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="transfer leader from store 14 to store 13"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.275 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="transfer leader from store 14 to store 13"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.277 +08:00] [Info] [cluster.go:531] ["leader changed"] [region-id=89472041] [from=14] [to=13]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.277 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 14"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.277 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 14"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.278 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 14"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.278 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 14"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.279 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89472042 store_id:14 }"] [old-confver=2110] [new-confver=2111]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.279 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476816 on store 87185011"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.281 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Add peer:{id:89476816 store_id:87185011 is_learner:true }"] [old-confver=2111] [new-confver=2112]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:49.281 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476816 on store 87185011"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.282 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="promote learner peer 89476816 on store 87185011 to voter"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.284 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89476816 store_id:87185011 is_learner:true },Add peer:{id:89476816 store_id:87185011 }"] [old-confver=2112] [new-confver=2113]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.284 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 5054383"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.284 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 5054383"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.286 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89472043 store_id:5054383 }"] [old-confver=2113] [new-confver=2114]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.286 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476814 on store 87185002"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.287 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Add peer:{id:89476814 store_id:87185002 is_learner:true }"] [old-confver=2114] [new-confver=2115]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.288 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="add learner peer 89476814 on store 87185002"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.665 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="promote learner peer 89476814 on store 87185002 to voter"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.678 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89476814 store_id:87185002 is_learner:true },Add peer:{id:89476814 store_id:87185002 }"] [old-confver=2115] [new-confver=2116]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.679 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="transfer leader from store 13 to store 17"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.680 +08:00] [Info] [cluster.go:531] ["leader changed"] [region-id=89472041] [from=13] [to=17]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.683 +08:00] [Info] [operator_controller.go:615] ["send schedule command"] [region-id=89472041] [step="remove peer on store 13"] [source=heartbeat]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.683 +08:00] [Info] [operator_controller.go:492] ["operator removed"] [region-id=89472041] [takes=2.434188312s] [operator="\"scatter-region {mv light peer: store [14 5054383 13] to [17 87185011 87185002]} (kind:leader,region, region:89472041(2223167,2108), createAt:2020-07-17 09:44:48.248532068 +0800 CST m=+40924.977299676, startAt:2020-07-17 09:44:48.24907874 +0800 CST m=+40924.977846330, currentStep:10, steps:[add learner peer 89476815 on store 17, promote learner peer 89476815 on store 17 to voter, transfer leader from store 14 to store 13, remove peer on store 14, add learner peer 89476816 on store 87185011, promote learner peer 89476816 on store 87185011 to voter, remove peer on store 5054383, add learner peer 89476814 on store 87185002, promote learner peer 89476814 on store 87185002 to voter, transfer leader from store 13 to store 17, remove peer on store 13])\""]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.683 +08:00] [Info] [operator_controller.go:567] ["operator canceled"] [region-id=89472041] [takes=2.43430889s] [operator="\"scatter-region {mv light peer: store [14 5054383 13] to [17 87185011 87185002]} (kind:leader,region, region:89472041(2223167,2108), createAt:2020-07-17 09:44:48.248532068 +0800 CST m=+40924.977299676, startAt:2020-07-17 09:44:48.24907874 +0800 CST m=+40924.977846330, currentStep:10, steps:[add learner peer 89476815 on store 17, promote learner peer 89476815 on store 17 to voter, transfer leader from store 14 to store 13, remove peer on store 14, add learner peer 89476816 on store 87185011, promote learner peer 89476816 on store 87185011 to voter, remove peer on store 5054383, add learner peer 89476814 on store 87185002, promote learner peer 89476814 on store 87185002 to voter, transfer leader from store 13 to store 17, remove peer on store 13])\""] [reason="cannot remove leader peer"]
pd_10_131_10_244_2379.log:[2020/07/17 09:44:50.684 +08:00] [Info] [cluster.go:519] ["region ConfVer changed"] [region-id=89472041] [detail="Remove peer:{id:89472044 store_id:13 }"] [old-confver=2116] [new-confver=2117]
```

### Describe the feature you'd like

expect the operator can be self re-build and continue to achieve the original purpose, as the above show, maybe like: transfer leader to the another peer and then remove the peer.

### Teachability, Documentation, Adoption, Migration Strategy

- Go