scylladb / scylladb/alternator-client-python

Coordinator-only nodes (server-side load balancing)

Open
#54 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
4
Avg merge
2h 49m
Merged PRs (30d)
7

Description

Jira task: https://scylladb.atlassian.net/browse/DRIVER-680
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-29

Copied from Jira epic DRIVER-29:

The original Alternator Load Balancing design document (https://docs.google.com/document/d/1twgrs6IM1B10BswMBUNqm7bwu5HCm47LOYE-Hdhuu_8/edit) suggested also the direction of server-side load balancing for unmodified clients. One of the implementations suggested was frontend (coordinator-only) nodes. Basically the idea is to have a regular Alternator node without data (owning zero tokens) serve as a load balancer to the other nodes that contain the data.

The purpose of this issue is to start testing and documenting this approach. I want to do the following:

  1. Write a test, perhaps in scylla.git's test/topology_custom (not here), that zero-token nodes actually work. We want to check it works for both tablets and vnodes-based tables, because currently Alternator defaults to using vnodes (but we can easily create an Alternator table with tablets for testing).
  2. Perform a very rudimentary performance test. The goal of this test isn't to measure absolute performance (so it won't need any nice beefy VMs) but just to measure relative performance between coordinator-only and data-holding nodes: if a data-holding Alternator nodes can finish N requests per second, we want the coordinator to be able to handle significantly more than N requests per second, otherwise it won't be useful as a load balancer (or, if we take larger nodes, this will be a very expensive solution)... In other words if coordinator is added to a K-node cluster (K=3,4,...), can we load the coordinator to 100% CPU (and leaving the cluster partially unused), or is the coordinator able to drive enough requests until the cluster is 100% used?
  3. Document in this repository how to set up the coordinator-only node and lessons we learned about it above. Start also thinking what we can do for high-availability (the coordinator node suddenly failing). Virtual IP address? DNS?

The relevant Scylla feature, coordinator-only nodes a.k.a. zero-token nodes or proxy nodes, was requested in https://github.com/scylladb/scylladb/issues/6527 and implemented in https://github.com/scylladb/scylladb/pull/19684.

CC @swasik

Migrated from GitHub issue: https://github.com/scylladb/alternator-load-balancing/issues/51

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the zero-token-node work referenced in scylla.git's test/topology_custom and the linked Scylla feature. Define tests for both tablets and vnodes, then compare rudimentary coordinator-only and data-node throughput. Done means the behavior is tested, performance observations are recorded, and this repository documents setup, lessons learned, and high-availability considerations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, documentation, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.