Recover with half of the coordinators
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Some regions from various cloud providers only provide two failure domains. In those regions we would like to guarantee that the failure of one region never causes data loss and that we could manually force a recovery with only half of the machines present. However, Coordinators currently prevent this from working.
It should be possible however to add an `fdbcli` command to force recovery. If one is running with an even number of coordinators and half of them are available, they are guaranteed to have a copy of the most recent cstate.
`fdbcli` could query all coordinators, grep the most recent cstate and move it to a new set of coordinators (to make sure the old coordinators can not rejoin). This has to be invoked manually and can't be done automatically otherwise a network partition could cause two fdb clusters to recover - so a human has to verify that this recovery is ok.
Contributor guide
Assessment
This issue has not been assessed yet.