apple / apple/foundationdb

Fault tolerance of the performant restore

Open
#2,948 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The performant restore to be released in FDB 6.3 assumes that no restore worker can fail.

This assumption is reasonable because (1) in a strictly controlled cluster, process and hardware failure is rare. If restore can finish within a short time, it will not need to handle failure; (2) removing failure help reduce the software complexity and deliver the feature earlier; (3) if failure happens in a restore, it can simply restart.

This assumption is not ideal because (1) in a cluster that is not fully controlled by DBA, the hardware can have regular maintenance. Not allowing failure will add strict constraint on the maintenance schedule; (2) it will prevent the performant restore from running continuously to check the backup's sanity; (3) it increases complexity for DBA to operate the restore cluster.

We plan to add fault tolerance to the performance restore after it has reached the desired performance.

Based on the current architecture of the performant restore, adding fault tolerance should not be super hard, although not easy.

**TODOs for supporting fault tolerance in the performance restore**:
- [ ] **Failure detection**: Use the destination FDB cluster's coordinators to monitor the restore master; Use the restore master to monitor the restore workers. It is similar to how FDB handles failures before FDB 6.3.
- [ ] **Progress checkpoint**: Restore master checkpoints the progress of each version batch; each restore applier saves the key-ranges it has applied to DB in each version batch. The checkpoint data is saved in the special system key space in the destination DB;
- [ ] **Recovery**: At recovery, restore master needs to (1) pause/terminate any ongoing work on restore workers; (2) figure out the restore point from the checkpoints; (3) restore the left-over progress in the ongoing version batches and the version batches after that. Note: Recovery should try to avoid the same failure happen again in the next epoch.

Contributor guide

Open the contributing guide

Research direction

Start by reading the current performant restore architecture, focusing on the restore master, restore workers, and destination-cluster coordinators. Map how workers report progress and how version batches are applied before designing failure detection, system-key-space checkpoints, and recovery. Done means failed components can be detected, progress can be recovered, and ongoing work resumes without repeating the same failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.