borgbackup / borgbackup/borg

Error "file inode changed" on ZFS snapshot

Open
#6,652 17 comments 2 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes. #6650 is about the same problem, but I'm not backing up a network filesystem.

## Is this a BUG / ISSUE report or a QUESTION?

~BUG~ QUESTION

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).
borg 1.2.0

#### Operating system (distribution) and version.
Ubuntu 20.04.4 LTS

#### Hardware / network configuration, and filesystems used.
Core i3 7320
2x 16GB Kingston DDR4-2400 ECC RAM
Filesystem for backup source: ZFS (snapshot)
Filesystem for backup target: ext4
Backup is not sent over network

#### How much data is handled by borg?
Around 2 TB per archive, around 152 TB in repository (2.5 TB deduplicated)

#### Full borg commandline that lead to the problem (leave away excludes and passwords)
```
BORG_RELOCATED_REPO_ACCESS_IS_OK=yes borgbackup create --compression lz4 --verbose --stats ${PATH_BACKUP}::$(/bin/date '+%Y-%m-%d_%H-%M') \
/root/ \
/storage/iobroker/.zfs/snapshot/borg/
// ... + a few more ZFS snapshot locations
```

## Describe the problem you're observing.

**UPDATE:** As it turns out that is probably intentional behaviour by ZFS. While investigating this I found out, that when accessing snapshots via the hidden .zfs directory, a temporary mount is created. That leads to the error described below, because the inode changes between the unmounted and the mounted snapshot.

My fix is to not rely on the automount feature, but instead mount the snapshot manually. I changed my backup script to mount all snapshots I want to backup into `/mnt/zfs/` and the problem is fixed.

Maybe this info helps someone that's why I didn't delete the issue, but changed it to be a question.

**Original Text:**
As far as I can see starting with borgbackup 1.2.0 the backup keeps failing with "file inode changed" errors:
```
Creating archive at "/mnt/backup_borg/files::2022-04-25_10-12"
/storage/backup/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/iobroker/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/minecraft/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/music/.zfs/snapshot/borg: file inode changed (race condition), skipping file
/storage/scans/.zfs/snapshot/borg: file inode changed (race condition), skipping file
------------------------------------------------------------------------------
Repository: /mnt/backup_borg/files
Archive name: 2022-04-25_10-12
Archive fingerprint: 9e3e209c74de5a120eeccb34d301da677f2d6eb3c1fb95c5c930910fe8ddb7b5
Time (start): Mon, 2022-04-25 10:12:57
Time (end): Mon, 2022-04-25 10:15:47
Duration: 2 minutes 49.51 seconds
Number of files: 1016125
Utilization of max. archive size: 0%
```

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

~Yes. It doesn't happen for all snapshots on very invocation as zfs seems to recycle inodes. But for each invocation it happens for 2 to 6 of the 10 snapshots I'm backupping.~

#### Include any warning/errors/backtraces from the system logs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.