apple / apple/foundationdb

Recover with half of the coordinators

Open
#2,914 6 comments 0 reactions 1 assignee Assigned to @sfc-gh-dyoungworth View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Some regions from various cloud providers only provide two failure domains. In those regions we would like to guarantee that the failure of one region never causes data loss and that we could manually force a recovery with only half of the machines present. However, Coordinators currently prevent this from working.

It should be possible however to add an `fdbcli` command to force recovery. If one is running with an even number of coordinators and half of them are available, they are guaranteed to have a copy of the most recent cstate.

`fdbcli` could query all coordinators, grep the most recent cstate and move it to a new set of coordinators (to make sure the old coordinators can not rejoin). This has to be invoked manually and can't be done automatically otherwise a network partition could cause two fdb clusters to recover - so a human has to verify that this recovery is ok.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.