Recovery without making the entire cluster unavailable
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
In current FDB, a tLog failure causes recovery. No transaction is allowed during this recovery time.
Since a tLog is only responsible for a set of key ranges, not the entire key range, can we allow transactions that will not touch the failed/limping tLog while the cluster is fixing the tLog?
It means we may want finer-granularity recovery: Instead of recovering everything in transaction system, we can divide it into smaller components and recover them independently.
In other words, it is like we are sharing the transaction systems, and allow recovery of each "shard".
For the commit, we still uses the same model, instead of using two-phase commit at these shards' level.
**A simple idea**:
Proxies just reject transactions that have mutations in the in-recovery shard and allow the rest of transactions.
**Challenges**:
- [ ] What if another failure happens while the cluster is recovering the first failed/limping tLog?
- [ ] If multiple components need recovery during a short period of time, how to coordinate them? (Maybe we should switch back to the current recovery model in this situation.)
...
**Visibility**:
We may also need to notify users about the mini-recovery so that they can change their transactions to avoid the shard that is in recovery.
Acknowledgement: This is from a discussion with Alex Shraer.
Contributor guide
Research direction
No implementation files, entry points, or tests are named. Start by mapping the transaction system, tLog recovery path, and proxy behavior, then review how failures and recovery are coordinated. Done would require a settled design for shard-level recovery, concurrent failures, coordination, and user visibility, with scope and validation criteria agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100