apple / apple/foundationdb

Investigate strategies to make Cluster Controller misplacement less likely

Open
#1,787 2 comments 0 reactions 1 assignee Claimed by @mpilman View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We have a feature that allows us to set a role for cluster controller. However, the way leader election works means that is not possible to guarantee that one of those processes will win the election.

This can be somewhat problematic: if a storage node wins leader election, the additional load from the cluster controller load might tip it over the edge. Worst case resolving this issue needs human interaction which is problematic given the number of production clusters we are running.

I propose a change to the coordinators to make happening this less likely:

- Add a new command line argument `--expect-cluster-controller-role` that the coordinators use
- If this flag is set, change the leader election like this:
1. immediately elect a leader if a cluster controller tried to become leader (and don't wait for more candidates)
1. Wait longer (for example 4 seconds) for candidates as long as no candidate has the cluster controller role.

This should make leader election faster in most cases and would reduce the probability for misplacing the CC.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.