[FR] File-Level Incremental Backup to Non-Btrfs Targets
- Dominant language
- Perl
- Stars
- 2.1k
- Forks
- 139
- PR merge metrics
- No merged PRs in 30d
Description
Currently, btrbk supports backups only to btrfs receive targets or raw stream files. There is no built-in option for performing file-level backups to non-btrfs destinations. While one could use rsyncon the latest snapshot, this requires a full filesystem scan each time and does not fully leverage the incremental capabilities of btrfs snapshots.
To address this, I propose enhancing btrbk with the following:
1. **Improve `btrbk diff`** to include deleted files, potentially using an approach similar to:
`btrfs send --no-data -p PARENT CHILD | btrfs receive --dump` (as discussed in #423).
2. **Use the output of `btrbk diff` to drive `rsync`**, so that only changed files are scanned and transferred. This would enable true incremental file-level backups to remote or non-btrfs storage.
These changes would allow efficient, snapshot-aware file-level backups without requiring a btrfs-compatible target. Other issues that may be related: #495, #630, #474
Backgroud
I manage several Ubuntu servers and back up their root filesystems to a TrueNAS server with ZFS. Previously, with ext4 rootfs, I used `rsnapshot` with hard links and rsync. However, this method lacks true atomic snapshots and requires scanning all files on every run.
After migrating to btrfs for atomic snapshots, I began using `btrbk`. The local snapshot feature works well, but remote backup is challenging in my setup:
- I cannot use `btrfs send ... | zfs receive ...` (hope we can one day?)
- TrueNAS does not include `btrfs-progs`, so a `btrfs receive` target is not feasible.
- Raw stream files stored via NFS are not directly readable and may not benefit from ZFS deduplication.
- A simple `rsync` from the latest snapshot remains a suboptimal workaround due to full scans.
While the proposed workflow could be scripted externally, e.g. by some pipelining like `ls ... | sort ... | btrfs send --no-data ... | btrfs receive --dump | grep ... | cut ... | sort ... | uniq ... | rsync ...`, integrating it into btrbk would offer a more streamlined and maintainable solution. Any other suggestions or solutions are welcomed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the btrbk diff workflow and the related discussion in #423, then review how its output could connect to rsync and btrfs send --no-data. Compare related issues #495, #630, and #474. Done would mean a defined, maintainable workflow for incremental file-level backups to non-btrfs targets, including deleted files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- perl
- Domain
- cli, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100