A simple algorithm for fully utilizing bandwidth for scaling in

## Feature Request

### Describe your feature request related problem

Following is the IO utilization in an experiment that generating snapshot is optimized to cost short constant time and 0 bandwidth.

The bandwidth used for snapshot replication is set to 50MiB/s. And sending snapshot is counted with factor 1/2 as the data may be in cache already, receiving snapshot is counted with factor 1 as data always have to be written to disk. There are four nodes, and one is being deleted. An optimal scheduling algorithm should utilize all bandwidth, that is 50 * 4 = 200MiB/s. So receiving snapshot should take about 200 * 2 / 3 = 133MiB/s. But the graph above shows that receiving snapshot can only fully utilize the bandwidth at some time.

### Describe the feature you'd like

A simple algorithm to fully utilize bandwidth is to **make every node both sending and receiving snapshot at the same time**. Just making one node keep sending multiple snapshots without receiving may not be optimal as the they may sending to the same other node, so the bandwidth utilization can only be 50 / 2 = 25MiB/s.

Based on the advice from #4099, we may not be able to let offline node to send or receive snapshots as all leaders are evicted. But thanks to https://github.com/tikv/raft-rs/issues/135, we can still make offline nodes keep sending snapshots. So if #4099 is implemented, the we can simply make offline stores send snapshots to all up nodes, and every up nodes should keep sending and receiving snapshots at the same time. I have developed the idea as a simple script and I can see the bandwidth is constantly at limit before offline store is about to get tombstone.

Note we don't need high store limit to fully utilize bandwidth. In my scripts, the store limit for every up nodes is 2.

A complicated way to fully utilize bandwidth is to setup a model to predict bandwidth at each node at real time.

### Describe alternatives you've considered
None

### Teachability, Documentation, Adoption, Migration Strategy
None