Use data distribution to limit aggregate write load to a storage server

While we currently support shard splitting to limit the amount of write bandwidth to a single shard, we don't track and manage the total write bandwidth to a single storage server. If a single storage server is responsible for multiple warm shards, it could in aggregate be hot and unable to do anything about it.

As an example, we suspect this could happen if there exist processes in the cluster which are responsible for a smaller amount of data than everybody else. If there are also hot shards in the cluster, they will get split and have a higher chance of moving to the processes that store less data. Done enough times, we could end up with a storage server that has a high aggregate bandwidth.

The proposal then is to track how much total write load each storage server has and move or split shards to mitigate spots where load is high.

Start by studying the existing shard-splitting behavior described in the issue and how storage servers are assigned multiple warm shards. Define how total write load would be tracked and how overloaded servers would trigger shard movement or splitting; done means aggregate write hotspots are mitigated without exceeding server capacity.

Written by the indexing model from the issue text.