hiero-ledger / hiero-ledger/hiero-consensus-node
Improve upload script to copy exisiting files instead of uploading them
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
Previously, the compaction mechanism updated every index file during the compaction. After implementation of #7501 ( [see details here](https://www.notion.so/swirldslabs/Compaction-Improvements-247726614d924fbaa34aa82a157a2f20 ), a majority the index of files will remain unchanged from round to round. The backup mechanism that we have can benefit from it.
The current backup script is straightforward - it takes a directory of a second latest snapshot and uploads it to GCP bucket.
Instead it can copy a previous backup using `rclone copy` and then run `rclone sync` to upload only the files that changed and delete the files that are no longer there.
[This is the scripts](https://github.com/swirlds/infrastructure/blob/master/containers/backup-uploader/backup.sh) that needs an update.
I believe that this change would be pretty safe:
1. In the worst case scenario (if the previous backup is absent or corrupted for some reasons) we'll upload the whole new backup which is fine. Right now we're doing it anyway.
2. We have the state validation tool that tests every backup for its correctness and fullness. If anything goes wrong, we'll notice it within 15 minutes.
Contributor guide
Research direction
Start by reading containers/backup-uploader/backup.sh and the current snapshot-to-GCP upload flow. Review the state validation tool before changing the backup steps. Done means a prior backup is copied with rclone copy, rclone sync uploads changed files and removes obsolete files, and validation confirms the backup remains complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, shell
- Domain
- cloud, devops
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100