If all replicas are lost, a StorageServer could source a shard replication from a backup.

If all replicas of a shard or set of shards are lost, it is actually possible, but slow, to restore them from an active backup.

Note that this plan assumes that although the shard is not readable it is still possible to commit blind writes to it. If we remove this requirement then the complexity is greatly reduced.

The sequence is roughly

1. Use the backup metadata to find the Key-Value Range Snapshot files relevant to the target set of shards and load the relevant ranges from those files.
2. Use the backup log stream to update each of the loaded ranges to a version which still exists in the FDB log system for the target shards.
3. Switch to using the FDB log system as the source of mutations (1-2 minutes behind, see below)
4. Keep applying until caught up.

There are of course a lot of details being glossed over here. Here are the ones I can think of:

- To make the restore efficient, Data Distribution should assign a single StorageServer with as many missing (0 replicas) shards as possible so that one restore can handle them all, since every execution of this process must read the entire mutation log from the snapshot versions to current. It should be possible to use the current average shard size to make an educated guess at this.
- The mutations for the shards being restored will be in the mutation tag stream for the StorageServer loading the data from backup, but they should not be applied when read from the log system.
- The StorageServer must still pop its tag from the log system after applying mutations (and ignoring the mutations destined for the shards being loaded) but it should pop on a delay of probably 1-2 minutes. This lag provides overlap with the most recently flushed/durable backup data so that the restore can switch from the backup log stream to the log system log stream which of course reaches all the way to the present version.

If instead blind writes to the lost shards is not allowed, then there is no need to switch to the log system as a mutation source in the context of the restore process. Once the backup mutation log has been used to update the shards to a data version at or greater than the point where each shard, respectively, was lost, then the shard can be brought back online.

Also, without the writability requirement then it could be argued that a separate selective restore using the existing process is the route to take. That's up for debate, but I rather like the elegance that DataDistribution could start this process automatically after shards are missing for some time, using the active backup on the default tag, and then could cancel the process if any of the shard replicas come back online.

No files, tests, or entry points are named. Start by mapping the Data Distribution and StorageServer restore flow, backup metadata, Key-Value Range Snapshot files, backup log stream, and FDB log system; resolve the blind-write requirement and choose between automatic restoration and selective restore. Done means the recovery behavior, mutation handling, lag, cancellation, and shard handoff are specified.

Written by the indexing model from the issue text.