apple / apple/foundationdb

Restore can fail to start for large backups on slow locally mounted filesystems

Open
#3,057 4 comments 0 reactions 1 assignee Claimed by @sfc-gh-dyoungworth View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

The BackupContainer implementation for locally mounted filesystems uses `platform::findFiles()` which uses blocking calls for opening and reading directories. This assumes the underlying filesystem will be very fast and can list the full directory tree in a short time, but this may not be the case for a large directory tree and a remote filesystem mounted locally.

The first task of a restore, executed by the backup agents, is to list all of the files in the backup to build a queue in the database. If the `findFiles()` call (link below) blocks for longer than the TaskBucket task timeout (something like 60 seconds) then the backup agent executing the file listing will lose its lease on the task and will fail.

https://github.com/apple/foundationdb/blob/release-6.2/fdbclient/BackupContainer.actor.cpp#L1325

To fix this, we need an async version of `platform::findFiles()` which is an actor and yields after calls to `opendir()` and `readdir()` which are both blocking. The existing version is fine for most of its use cases, so to avoid code duplication without breaking existing code I suggest
- convert `platform::findFiles()` to an actor
- add a required argument `bool async` which indicates to yield after calls to blocking functions
- create an overload/wrapper of `findFiles()` with the original signature which is a drop in replacement for all existing use cases - it is not an actor and returns `findFiles(..., false).get()` which is fine since the false means `findFiles()` will never wait.
- add a convenience wrapper called `findFilesAsync()` which returns `findFiles(..., true)`

Note that if for some reason a single call to `opendir()` or `readdir()` takes excessively long then a restore start could still fail due to task lease timeout.

Note that BackupContainerBlobStore does not have this issue, its I/O is not blocking so the executing agent can renew its task lease while waiting on delayed responses.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.