hiero-ledger / hiero-ledger/hiero-consensus-node

Improve upload script to copy exisiting files instead of uploading them

Open
#9,825 0 comments 0 reactions 0 assignees View on GitHub
Platform
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Previously, the compaction mechanism updated every index file during the compaction. After implementation of #7501 ( [see details here](https://www.notion.so/swirldslabs/Compaction-Improvements-247726614d924fbaa34aa82a157a2f20 ), a majority the index of files will remain unchanged from round to round. The backup mechanism that we have can benefit from it.

The current backup script is straightforward - it takes a directory of a second latest snapshot and uploads it to GCP bucket.

Instead it can copy a previous backup using `rclone copy` and then run `rclone sync` to upload only the files that changed and delete the files that are no longer there.

[This is the scripts](https://github.com/swirlds/infrastructure/blob/master/containers/backup-uploader/backup.sh) that needs an update.

I believe that this change would be pretty safe:
1. In the worst case scenario (if the previous backup is absent or corrupted for some reasons) we'll upload the whole new backup which is fine. Right now we're doing it anyway.
2. We have the state validation tool that tests every backup for its correctness and fullness. If anything goes wrong, we'll notice it within 15 minutes.

Contributor guide

Open the contributing guide

Research direction

Start by reading containers/backup-uploader/backup.sh and the current snapshot-to-GCP upload flow. Review the state validation tool before changing the backup steps. Done means a prior backup is copied with rclone copy, rclone sync uploads changed files and removes obsolete files, and validation confirms the backup remains complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, shell
Domain
cloud, devops
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.