Proposal for rotation of raw incremental backups
- Dominant language
- Perl
- Stars
- 2.1k
- Forks
- 139
- PR merge metrics
- No merged PRs in 30d
Description
I propose a related set of features to enable automatic rotation (deletion) of incremental raw backups.
1. **Honor `target_preserve`, additionally preserving ancestors.** EDIT: already implemented. ~~`btrbk` should preserve backups as specified by the user, *as well as* the chain of backups required to construct the desired preserved backups. `btrbk` can walk the tree of `RECEIVED_PARENT_UUID` relationships given by `*.info` files to determine which additional backups need to be preserved. AFAICT `btrbk` already has some logic to determine this, but it's not used (?), since the preserve mechanism for raw targets isn't used.~~
2. **Provide a way to balance the ancestor tree, e.g. an explicit schedule configuration in place of `incremental_prefs`.** Currently the parent of each backup is usually the previous backup, so the ancestor tree is just a long chain, and preserving ancestors would mean preserving everything. We need to shorten and balance this tree to have meaningful backup rotation. Tree management is a broad topic, but it would be useful to start with some initial approach (of course, the one that works for my use case :smile:) and expand later.
3. Already implemented, but we should document that incremental raw backups only use a single parent (`-p`, but not `-c`). It would be nice to document the trade-off here: multiple parents would increase risk that a broken backup would break more descendant backups, and would make backups harder to understand, and the only benefit of `-c` is saving data storage in the case of cross-snapshot reflinks, which are probably rare.
As a starting point for tree management, I suggest a schedule config for incremental parents, similar to retention policies. The concept is that each level of the ancestor tree corresponds to one of the yearly/monthly/weekly/daily/hourly schedule concepts, which are already well-defined in btrbk.
`incremental_schedule 1y 1m 1d` could mean that:
* A yearly backup has no parent (full backup)
* A monthly backup uses the latest yearly backup as its parent
* A daily backup uses the latest monthly backup as its parent
* All other backups (hourly, ad-hoc, etc) default to using the latest backup as their parent
# My use case
I have a large (300gb compressed) homedir in which I do regular daily work, but most of the data rarely changes. I back up to Backblaze B2 via s3fs. I use a 90-day object retention policy for safety.
I don't want to do full backups very often, to minimize disk/network/cpu usage, and because object retention means retaining a large backup as well as its nearly-identical replacement, whenever I do a full backup.
I want to back up as frequently as possible.
(5-second backup intervals would be ideal. Why? Because it's $YEAR and I expect my work to always be backed up (think Google Docs). Because we as a species have already done the hard work to develop btrfs, which supports arbitrarily many atomic snapshots, `onchange` semantics, and easy deltas. Because backblaze doesn't charge per-object fees. The hard parts are all solved, and backup software shouldn't stop me from making use of them. That said, s3fs' caching is poor and issues `HeadObject` requests too often, so my backups are more infrequent to lower active object counts, and my dreams are once more dashed by one piece of bad software)
I would use a config like this:
```
volume /mnt/btr_pool
subvolume home
target raw /s3
incremental_schedule 1y 1m 1d # proposed new config
target_preserve 30d
target_preserve_min no
snapshot_preserve 24h
snapshot_preserve_min latest
```
* The first backup of the year will be a full backup
* The first backup of each month will use the yearly backup as a parent
* The first backup of each day will use the last monthly backup as its parent
* The last 30 daily backups will be preserved, but ancestor preservation means the yearly backups *and snapshots* will be preserved too. In particular:
* On 2023-01-15, we'll preserve:
* Backups from 2022-12-16 through 2023-01-15, by `target_preserve`
* Backups for 2022-12-01 and **2022**-01-01, by ancestor preservation
* The snapshot for 2023-01-01, by `incremental_schedule`, as we'll need it as the parent of tomorrow's daily backup
* On 2023-06-15, we'll preserve:
* Backups from 2023-05-15 through 2023-06-15, by `target_preserve`
* Backups for 2023-05-01 and 2023-01-01, by ancestor preservation
* The snapshot for 2023-06-01, by `incremental_schedule`, as we'll need it as the parent of tomorrow's daily backup
* The snapshot for 2023-01-01, by `incremental_schedule`, as we'll need it as the parent of next month's monthly backup
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.