apple / apple/foundationdb

Recover from a cluster with no coordinated state

Open
#2,022 12 comments 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I know of three cases so far where someone deleted the coordinated state and therefore lost all of their data. I think there are some things we can do to make this better:

1. If `exclude` is used to exclude coordinators I think we should fail or at least warn. Exclude doesn't do what most people expect it will do for coordinators.
1. We should provide a tool that helps a user to recover from this situation.

One idea for (2) is:

We could write the coordinated state into each tlog (we could do this with the recovery transaction). We later could provide a tool that would fetch this state from all tlogs and would try to reconstruct a valid coordinated state. I think we could even do this without data loss - but even if we would lose some mutations it would be better than recovering from a backup.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by understanding how coordinator exclusion, coordinated state deletion, recovery transactions, and tlogs currently work. Done would require an agreed recovery design, including whether exclusion warns or fails and how a tool reconstructs coordinated state.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.