Investigate strategies to make Cluster Controller misplacement less likely
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
We have a feature that allows us to set a role for cluster controller. However, the way leader election works means that is not possible to guarantee that one of those processes will win the election.
This can be somewhat problematic: if a storage node wins leader election, the additional load from the cluster controller load might tip it over the edge. Worst case resolving this issue needs human interaction which is problematic given the number of production clusters we are running.
I propose a change to the coordinators to make happening this less likely:
- Add a new command line argument `--expect-cluster-controller-role` that the coordinators use
- If this flag is set, change the leader election like this:
1. immediately elect a leader if a cluster controller tried to become leader (and don't wait for more candidates)
1. Wait longer (for example 4 seconds) for candidates as long as no candidate has the cluster controller role.
This should make leader election faster in most cases and would reduce the probability for misplacing the CC.
Contributor guide
Assessment
This issue has not been assessed yet.