Node-size awareness (for token-unaware load balancing)

This issue is about the client-side load balancer needing to send fewer requests to smaller nodes than to bigger nodes.

Although in the past we recommended that a Scylla cluster should contain many identically-sized nodes, we've recently reversed this recommendation:

1. With tablets, it's easy and natural to support different-sized nodes. A half-sized node to get half the tablets - and therefore half the storage and half the work - of a double-sized node.
2. Different-sized nodes allow much finer granularity in cluster growth: If we have a 3-node cluster, one node on each rack, and want to grow it by adding same-sized nodes, the minimum growth we can achieve is doubling the cluster to 6 nodes. If we allow adding a smaller node to each rack, we can grow the cluster at smaller increments.

If we implement token-aware load balancing, we will support the different-sized-nodes case automatically: Smaller nodes will have fewer tokens, i.e. fewer tablets, so will get fewer requests.

However, until token-aware load balancing exists, we need a different solution to support different-sized nodes.

What we can do is for the load balancing library to figure out the size of each node and instead of picking a node uniformly from the list of live nodes, give larger nodes a higher probability of being picked. We could expand the /localnodes response, perhaps if given a new optional parameter, to return the size of each node, or alternatively the load balancer could figure out the size of each node by other means, reading tablets system table or REST API. Perhaps can use the number of shards on a node as a good estimate of its strength. The number of tablets on each node is probably an even better criterion, if we assume that Scylla already tries its best to distribute tablets according to the strength of of the nodes - because our state-of-the-art tablet-aware CQL drivers will balance the load according to these tablets.

Jira:

• Driver issue: https://scylladb.atlassian.net/browse/DRIVER-555
• Parent epic: https://scylladb.atlassian.net/browse/DRIVER-42

Start with the /localnodes response described in the issue, then review DRIVER-555 and DRIVER-42 for the intended driver scope. Compare the proposed node-size signals, such as shards or tablets, and define how the client should weight live-node selection; done means the token-unaware load balancer sends proportionally fewer requests to smaller nodes.