Error "file inode changed" on ZFS snapshot
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
## Have you checked borgbackup docs, FAQ, and open Github issues?
Yes. #6650 is about the same problem, but I'm not backing up a network filesystem.
## Is this a BUG / ISSUE report or a QUESTION?
~BUG~ QUESTION
## System information. For client/server mode post info for both machines.
#### Your borg version (borg -V).
borg 1.2.0
#### Operating system (distribution) and version.
Ubuntu 20.04.4 LTS
#### Hardware / network configuration, and filesystems used.
Core i3 7320
2x 16GB Kingston DDR4-2400 ECC RAM
Filesystem for backup source: ZFS (snapshot)
Filesystem for backup target: ext4
Backup is not sent over network
#### How much data is handled by borg?
Around 2 TB per archive, around 152 TB in repository (2.5 TB deduplicated)
#### Full borg commandline that lead to the problem (leave away excludes and passwords)
```
BORG_RELOCATED_REPO_ACCESS_IS_OK=yes borgbackup create --compression lz4 --verbose --stats ${PATH_BACKUP}::$(/bin/date '+%Y-%m-%d_%H-%M') \
/root/ \
/storage/iobroker/.zfs/snapshot/borg/
// ... + a few more ZFS snapshot locations
```
## Describe the problem you're observing.
**UPDATE:** As it turns out that is probably intentional behaviour by ZFS. While investigating this I found out, that when accessing snapshots via the hidden .zfs directory, a temporary mount is created. That leads to the error described below, because the inode changes between the unmounted and the mounted snapshot.
My fix is to not rely on the automount feature, but instead mount the snapshot manually. I changed my backup script to mount all snapshots I want to backup into `/mnt/zfs/` and the problem is fixed.
Maybe this info helps someone that's why I didn't delete the issue, but changed it to be a question.
**Original Text:**
As far as I can see starting with borgbackup 1.2.0 the backup keeps failing with "file inode changed" errors:
```
Creating archive at "/mnt/backup_borg/files::2022-04-25_10-12"
/storage/backup/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/iobroker/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/minecraft/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/music/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/scans/.zfs/snapshot/borg: file inode changed (race condition), skipping file
------------------------------------------------------------------------------
Repository: /mnt/backup_borg/files
Archive name: 2022-04-25_10-12
Archive fingerprint: 9e3e209c74de5a120eeccb34d301da677f2d6eb3c1fb95c5c930910fe8ddb7b5
Time (start): Mon, 2022-04-25 10:12:57
Time (end): Mon, 2022-04-25 10:15:47
Duration: 2 minutes 49.51 seconds
Number of files: 1016125
Utilization of max. archive size: 0%
```
#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
~Yes. It doesn't happen for all snapshots on very invocation as zfs seems to recycle inodes. But for each invocation it happens for 2 to 6 of the 10 snapshots I'm backupping.~
#### Include any warning/errors/backtraces from the system logs
Contributor guide
Assessment
This issue has not been assessed yet.