apple / apple/foundationdb

Don't lose data if two availability zones go down

Open
#3,638 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

If `three_data_hall` is used across three availability zones, we currently guarantee that FDB can survive the failure of one AZ and one machine without any availability loss.

However, I would like to also be able to survive two AZ failure without any data loss (we would lose availability until one AZ comes back).

It is not quite clear to me how we can achieve this, but I think it would involve something like this:

1. During recruitment try to recruit tlogs in all availability zones for `X` seconds (`X` would be a configuration parameter)
2. If recruiting in three AZs is impossible, recruit only in two, but set failure tolerance to a different value.
3. Don't recover at all if only one AZ is available.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing three_data_hall recruitment, failure-tolerance configuration, and recovery behavior; done means defining and validating behavior for three, two, and one available availability zones without data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.