Recover from a cluster with no coordinated state
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
I know of three cases so far where someone deleted the coordinated state and therefore lost all of their data. I think there are some things we can do to make this better:
1. If `exclude` is used to exclude coordinators I think we should fail or at least warn. Exclude doesn't do what most people expect it will do for coordinators.
1. We should provide a tool that helps a user to recover from this situation.
One idea for (2) is:
We could write the coordinated state into each tlog (we could do this with the recovery transaction). We later could provide a tool that would fetch this state from all tlogs and would try to reconstruct a valid coordinated state. I think we could even do this without data loss - but even if we would lose some mutations it would be better than recovering from a backup.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by understanding how coordinator exclusion, coordinated state deletion, recovery transactions, and tlogs currently work. Done would require an agreed recovery design, including whether exclusion warns or fails and how a tool reconstructs coordinated state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100