apple / apple/foundationdb

Add support for multiple cluster controllers

Open
#1,013 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The cluster controller is a singleton role which can be overwhelmed if too many processes are connected to the database. This limits the total number of processes that can join a database, and therefore limits how large a database can scale.

In addition to scaling concerns, supporting multiple cluster controllers will enable a datacenter that is partitioned from the rest of the system to still provide stale reads from that location.

Finally, having multiple cluster controllers will reduce or eliminate the cost of starting up a new cluster controller when either a cluster controller has died, or we are switching the primary datacenter of a cluster. This should reduce the master recovery times in these scenarios.

Proposed design:
* Each region has its own local coordinators and local cluster controller. A region can have more than cluster controller, in which case processes in that region choose which cluster controller to register with arbitrarily.
* Workers register only with their local cluster controller.
* Clients only connect to their local cluster controller.
* The local cluster controllers use the global coordinators to elect one of them to be responsible for electing the master.
* Failure monitoring data is shared between cluster controllers.
* The list of registered workers is shared between cluster controllers.
* After a master is fully recovered, it will attempt to write the coordinated state to all of the local coordinators in each data center.
* The master registers with all cluster controllers, if a local cluster controller dies it must resend the registration to the newly elected local cluster controller.
* Tlog rejoins are processed by the local cluster controllers.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by mapping the existing cluster-controller, coordinator, worker-registration, master-recovery, and Tlog-rejoin paths against the proposed design. Done would require implementing and validating coordinated behavior across multiple local cluster controllers, including election, shared state, registration, recovery, and rejoins.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.