scylladb / scylladb/alternator-client-python
Node-size awareness (for token-unaware load balancing)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 4
- Avg merge
- 2h 49m
- Merged PRs (30d)
- 7
Description
Jira task: https://scylladb.atlassian.net/browse/DRIVER-556
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-42
Copied from Jira epic DRIVER-42:
This issue is about the client-side load balancer needing to send fewer requests to smaller nodes than to bigger nodes.
Although in the past we recommended that a Scylla cluster should contain many identically-sized nodes, we've recently reversed this recommendation:
- With tablets, it's easy and natural to support different-sized nodes. A half-sized node to get half the tablets - and therefore half the storage and half the work - of a double-sized node.
- Different-sized nodes allow much finer granularity in cluster growth: If we have a 3-node cluster, one node on each rack, and want to grow it by adding same-sized nodes, the minimum growth we can achieve is doubling the cluster to 6 nodes. If we allow adding a smaller node to each rack, we can grow the cluster at smaller increments.
If we implement token-aware load balancing ([#11|https://github.com/scylladb/alternator-load-balancing/issues/11]) , we will support the different-sized-nodes case authomatically: Smaller nodes will have fewer tokens (i.e., fewer tablets), so will get fewer requests.
However, we still haven't implemented [#11|https://github.com/scylladb/alternator-load-balancing/issues/11], so unless we do it soon we need a different solution to support different-sized nodes.
What we can do is for the load balancing library to figure out the size of each node and instead of picking a node uniformly from the list of live nodes, give larger nodes a higher probability of being picked. We could expand the /localnodes response (perhaps if given a new optional parameter) to return the size of each node, or alternatively the load balancer could figure out the size of each node by other means (reading tablets system table? REST API?). Perhaps can use the number of shards on a node as a good estimate of its strength. The number of tablets on each node is probably an even better criterion, if we assume that Scylla already tries its best to distribute tablets according to the strength of of the nodes - because our state-of-the-art tablet-aware CQL drivers will balance the load according to these tablets.
Migrated from GitHub issue: https://github.com/scylladb/alternator-load-balancing/issues/128
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the client-side load-balancer node-selection logic and the /localnodes response mentioned in the issue. Compare the proposed node-size signals—shards, tablets, or another size value—and define a solution that sends fewer requests to smaller nodes instead of selecting live nodes uniformly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100