internal/dave/repo_s3: handle upload failures
- Dominant language
- Go
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The default logic of Dave is:
1. Clone data directories to Dave home
2. Create archives of cloned data directories
3. Add snapshot containing archives to repository (uploading each archive to S3)
4. Remove local copies of archives to reduce storage requirement
The issue with this logic is that if a part of step 3 fails, Dave may delete all local copies of the archives because it is "done" with the archives. We have encountered cases where uploading very large archives to S3 is unstable, resulting in us needing to manually upload the archives with the `aws s3` CLI.
A `--keep-archives` flag was added in https://github.com/hemilabs/dave/pull/5/commits/37db1636f1ced19cc0988bbe0b76760807f56f65 as a temporary way to work around this issue and manually upload the archives.
We should improve how we handle these upload failures, making it possible to retry uploading remaining archives if Dave fails. Dave should also retry multiple times to continue uploading files to S3 if it encounters a temporary error.
Contributor guide
Assessment
This issue has not been assessed yet.