apple / apple/foundationdb

Recovery without making the entire cluster unavailable

Open
#3,270 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In current FDB, a tLog failure causes recovery. No transaction is allowed during this recovery time.

Since a tLog is only responsible for a set of key ranges, not the entire key range, can we allow transactions that will not touch the failed/limping tLog while the cluster is fixing the tLog?

It means we may want finer-granularity recovery: Instead of recovering everything in transaction system, we can divide it into smaller components and recover them independently.
In other words, it is like we are sharing the transaction systems, and allow recovery of each "shard".
For the commit, we still uses the same model, instead of using two-phase commit at these shards' level.

**A simple idea**:
Proxies just reject transactions that have mutations in the in-recovery shard and allow the rest of transactions.

**Challenges**:
- [ ] What if another failure happens while the cluster is recovering the first failed/limping tLog?
- [ ] If multiple components need recovery during a short period of time, how to coordinate them? (Maybe we should switch back to the current recovery model in this situation.)
...

**Visibility**:
We may also need to notify users about the mini-recovery so that they can change their transactions to avoid the shard that is in recovery.

Acknowledgement: This is from a discussion with Alex Shraer.

Contributor guide

Open the contributing guide

Research direction

No implementation files, entry points, or tests are named. Start by mapping the transaction system, tLog recovery path, and proxy behavior, then review how failures and recovery are coordinated. Done would require a settled design for shard-level recovery, concurrent failures, coordination, and user visibility, with scope and validation criteria agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.