scatter-region scheduler does not get re-created after leadership transfer
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
Scatter-region scheduler does not get re-created properly after PD leadership transfer.
### What did you do?
1. Start scatter region scheduler using TiDB API
2. `member leader resign`
3. Observe that the scheduler gets created, but all operators get cancelled immediately and no real region movements are done
### What did you expect to see?
Scatter-region scheduler should continue to scatter regions after PD leader changes.
### What did you see instead?
Sample logs from the new leader to demonstrate key range encoding/decoding issue after leadership transfer:
```
[2025/08/14 23:47:55.814 +00:00] [INFO] [coordinator.go:333] ["create scheduler"] [scheduler-name=scatter-range-scheduler-mytable-468] [scheduler-args="[\"t�\\u0000\\u0000\\u0000\\u0000\\u0000\\u0001��_i�\\u0000\\u0000\\u0000\\u0000�\\u0000\\u0000\\u0005\\u0000\\u0000\\u0000\\u0000\\u0000�\",\"t�\\u0000\\u0000\\u0000\\u0000\\u0000\\u0001��_i�\\u0000\\u0000\\u0000\\u0000�\\u0000\\u0000\\u0005�\\u0000\\u0000\\u0000\\u0000�\",mytable-468]"]
....
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=8] [operator="\"scatter-range-leader-mytable-468 {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809369905 +0000 UTC m=+277685.675473319, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=8] [operator="\"scatter-range-leader-mytable-468 {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809369905 +0000 UTC m=+277685.675473319, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=8] [operator="\"scatter-range-leader-mytable-468-anotherindex_role {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809423974 +0000 UTC m=+277685.675527387, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:510] ["add operator"] [region-id=8] [operator="\"scatter-range-leader-mytable-468-action {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809500352 +0000 UTC m=+277685.675603761, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:705] ["replace old operator"] [region-id=8] [takes=118.948µs] [operator="\"scatter-range-leader-mytable-468 {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809369905 +0000 UTC m=+277685.675473319, startAt:2025-08-14 23:47:56.80944936 +0000 UTC m=+277685.675552773, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.809 +00:00] [INFO] [operator_controller.go:725] ["operator canceled"] [region-id=8] [takes=0s] [operator="\"scatter-range-leader-mytable-468-PRIMARY {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.809594299 +0000 UTC m=+277685.675697707, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"cancel-reason\":\"already exist\",\"sourceScore\":\"760.00\",\"targetScore\":\"2.00\"}"]
[2025/08/14 23:47:56.810 +00:00] [INFO] [operator_controller.go:725] ["operator canceled"] [region-id=8] [takes=0s] [operator="\"scatter-range-leader-mytable-468-anotherindex_scope {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.810437224 +0000 UTC m=+277685.676540671, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"cancel-reason\":\"already exist\",\"sourceScore\":\"759.00\",\"targetScore\":\"3.00\"}"]
[2025/08/14 23:47:56.810 +00:00] [INFO] [operator_controller.go:725] ["operator canceled"] [region-id=8] [takes=0s] [operator="\"scatter-range-leader-mytable-468-status {transfer leader: store 36 to 12} (kind:range,leader, region:8(7, 699), createAt:2025-08-14 23:47:56.810505943 +0000 UTC m=+277685.676609388, startAt:0001-01-01 00:00:00 +0000 UTC, currentStep:0, size:0, steps:[0:{transfer leader from store 36 to store 12}], timeout:[1m0s])\""] [additional-info="{\"cancel-reason\":\"already exist\",\"sourceScore\":\"759.00\",\"targetScore\":\"3.00\"}"]
```
When the scatter scheduler is created from HTTP API, it is using a different key range:
```
[2025/08/15 23:48:03.375 +00:00] [INFO] [audit.go:126] ["audit log"] [service-info="{ServiceLabel:CreateScheduler, Method:HTTP/2.0/POST:/pd/api/v1/schedulers, Component:anonymous, IP:---, StartTime:2025-08-15 23:48:03 +0000 UTC, URLParam:{}, BodyParam:{\"end_key\":\"t%80%00%00%00%00%00%0E%FFB_r%FF%FF%FF%FF%FF%FF%FF%FF%FF%00%00%00%00%00%FA\",\"name\":\"scatter-range\",\"range_name\":\"mytable-3650\",\"start_key\":\"t%80%00%00%00%00%00%0E%FFB_r%00%00%00%00%00%FF%00%00%00%00%00%00%00%00%FA\"}}"]
[2025/08/15 23:48:03.375 +00:00] [INFO] [handler.go:228] ["create scheduler"] [scheduler-name=scatter-range-mytable-3650] [scheduler-args="[\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u000e\\ufffdB_r\\u0000\\u0000\\u0000\\u0000\\u0000\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\ufffd\",\"t\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\u000e\\ufffdB_r\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\ufffd\\u0000\\u0000\\u0000\\u0000\\u0000\\ufffd\",mytable-3650]"]
```
Although the correct log is from another cluster, the behavior is consistent. Due to the incorrect key range parsing, the scheduler kept thinking region 8 is a part of the table, while region 8 only hosts metadata tables like `information_schema` and `performance_schema`. The scheduler kept creating operator for region 8, but region 8 is already perfectly balanced, then somehow this blocks all the real region scattering work.
### What version of PD are you using (`pd-server -V`)?
v8.5.1
Contributor guide
Assessment
This issue has not been assessed yet.