region related operators should never timeout
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Bug Report
### What did you do?
Run a cluster with large regions
### What did you expect to see?
Store will get balance.
### What did you see instead?
PD randomly schedule regions around.
After changing region scheduler operator timeout to 1h, randomly scheduling seems to be fixed.
The default timeout is about 5 minutes. Supposing there are 8 concurrently region operators to transfer peer, and the replication rate limit is 100MiB. Even all bandwidth is utilized, 5 minutes can only deliver about 29.2GiB data. This means if the size of a region exceeds 3.6GiB or network/disk IO becomes slow, the operators can eventually timeout.
However, it's meaningless to timeout a region operator, it should either finish or aborted. If it times out, it means TiKV is doing a lot more data replications than PD expected, and disk usage is not correct anymore as snapshot can be applied at any time. It will run into chaos in the end.
### What version of PD are you using (`pd-server -V`)?
v5.1.1
Contributor guide
Assessment
This issue has not been assessed yet.