digint / digint/btrbk

Proposal for rotation of raw incremental backups

Open
#474 25 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Perl
Stars
2.1k
Forks
139
PR merge metrics
No merged PRs in 30d

Description

I propose a related set of features to enable automatic rotation (deletion) of incremental raw backups.

1. **Honor `target_preserve`, additionally preserving ancestors.** EDIT: already implemented. ~~`btrbk` should preserve backups as specified by the user, *as well as* the chain of backups required to construct the desired preserved backups. `btrbk` can walk the tree of `RECEIVED_PARENT_UUID` relationships given by `*.info` files to determine which additional backups need to be preserved. AFAICT `btrbk` already has some logic to determine this, but it's not used (?), since the preserve mechanism for raw targets isn't used.~~
2. **Provide a way to balance the ancestor tree, e.g. an explicit schedule configuration in place of `incremental_prefs`.** Currently the parent of each backup is usually the previous backup, so the ancestor tree is just a long chain, and preserving ancestors would mean preserving everything. We need to shorten and balance this tree to have meaningful backup rotation. Tree management is a broad topic, but it would be useful to start with some initial approach (of course, the one that works for my use case :smile:) and expand later.
3. Already implemented, but we should document that incremental raw backups only use a single parent (`-p`, but not `-c`). It would be nice to document the trade-off here: multiple parents would increase risk that a broken backup would break more descendant backups, and would make backups harder to understand, and the only benefit of `-c` is saving data storage in the case of cross-snapshot reflinks, which are probably rare.

As a starting point for tree management, I suggest a schedule config for incremental parents, similar to retention policies. The concept is that each level of the ancestor tree corresponds to one of the yearly/monthly/weekly/daily/hourly schedule concepts, which are already well-defined in btrbk.

`incremental_schedule 1y 1m 1d` could mean that:
* A yearly backup has no parent (full backup)
* A monthly backup uses the latest yearly backup as its parent
* A daily backup uses the latest monthly backup as its parent
* All other backups (hourly, ad-hoc, etc) default to using the latest backup as their parent

# My use case

I have a large (300gb compressed) homedir in which I do regular daily work, but most of the data rarely changes. I back up to Backblaze B2 via s3fs. I use a 90-day object retention policy for safety.

I don't want to do full backups very often, to minimize disk/network/cpu usage, and because object retention means retaining a large backup as well as its nearly-identical replacement, whenever I do a full backup.

I want to back up as frequently as possible.

(5-second backup intervals would be ideal. Why? Because it's $YEAR and I expect my work to always be backed up (think Google Docs). Because we as a species have already done the hard work to develop btrfs, which supports arbitrarily many atomic snapshots, `onchange` semantics, and easy deltas. Because backblaze doesn't charge per-object fees. The hard parts are all solved, and backup software shouldn't stop me from making use of them. That said, s3fs' caching is poor and issues `HeadObject` requests too often, so my backups are more infrequent to lower active object counts, and my dreams are once more dashed by one piece of bad software)

I would use a config like this:
```
volume /mnt/btr_pool
subvolume home
target raw /s3
incremental_schedule 1y 1m 1d # proposed new config
target_preserve 30d
target_preserve_min no
snapshot_preserve 24h
snapshot_preserve_min latest
```
* The first backup of the year will be a full backup
* The first backup of each month will use the yearly backup as a parent
* The first backup of each day will use the last monthly backup as its parent
* The last 30 daily backups will be preserved, but ancestor preservation means the yearly backups *and snapshots* will be preserved too. In particular:
* On 2023-01-15, we'll preserve:
* Backups from 2022-12-16 through 2023-01-15, by `target_preserve`
* Backups for 2022-12-01 and **2022**-01-01, by ancestor preservation
* The snapshot for 2023-01-01, by `incremental_schedule`, as we'll need it as the parent of tomorrow's daily backup
* On 2023-06-15, we'll preserve:
* Backups from 2023-05-15 through 2023-06-15, by `target_preserve`
* Backups for 2023-05-01 and 2023-01-01, by ancestor preservation
* The snapshot for 2023-06-01, by `incremental_schedule`, as we'll need it as the parent of tomorrow's daily backup
* The snapshot for 2023-01-01, by `incremental_schedule`, as we'll need it as the parent of next month's monthly backup

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.