tikv / tikv/pd

region related operators should never timeout

Open
#4,136 2 comments 0 reactions 0 assignees View on GitHub
status/discussion-wanted type/design
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Bug Report

### What did you do?
Run a cluster with large regions

### What did you expect to see?
Store will get balance.

### What did you see instead?
PD randomly schedule regions around.

After changing region scheduler operator timeout to 1h, randomly scheduling seems to be fixed.

The default timeout is about 5 minutes. Supposing there are 8 concurrently region operators to transfer peer, and the replication rate limit is 100MiB. Even all bandwidth is utilized, 5 minutes can only deliver about 29.2GiB data. This means if the size of a region exceeds 3.6GiB or network/disk IO becomes slow, the operators can eventually timeout.

However, it's meaningless to timeout a region operator, it should either finish or aborted. If it times out, it means TiKV is doing a lot more data replications than PD expected, and disk usage is not correct anymore as snapshot can be applied at any time. It will run into chaos in the end.

### What version of PD are you using (`pd-server -V`)?
v5.1.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.